REVIEW 3 major objections 5 minor 23 references
A Toolkit for Compliance, a Toolkit for Justice: Drawing on Cross-sectoral Expertise to Develop a Pro-justice EU AI Act Toolkit
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A year-long academic-industry co-design produced a free, open-access EU AI Act compliance toolkit that scaffolds iterative, justice-oriented reflection, and the process itself is offered as a blueprint.
desk verdict A real, open-access toolkit and an honest process narrative; the compliance mapping needs more evidence but the paper deserves review, not rejection. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the toolkit's 'spaces' structure: seven interconnected areas of focus (Business, Team and Principles; Impact, Risk and Mitigation; Stakeholder Engagement; User-Centred Design; Data Governance; Model Governance; Evaluation and Care) that teams revisit rather than complete in sequence. Each task within a space uses a fixed five-part scaffold (Description, Rationale and link to the Act, How to Approach, Expertise and Engagement, and Go Further) plus a workbook that records and exports documentation. This structure carries the argument by making iterative ethical deliberation concrete, documentable, and attached to specific legal obligations.
What would settle it
Compare the toolkit's task list against the full text of the EU AI Act's obligations for high-risk systems and identify any obligation that has no corresponding task, or have an independent legal audit find that completing all toolkit tasks still leaves a high-risk provider short of a documented compliance obligation.
Extended reading notes
Core claim
The paper claims that a pro-justice EU AI Act toolkit can be built and that the collaboration that built it is reusable. The toolkit organizes compliance into seven revisitable 'spaces' and guides product managers and their teams through tasks that explain, justify, and document AI Act obligations while adding voluntary considerations such as stakeholder engagement, disability justice, redress, and environmental impact. The authors report that the workflow deliberately avoids completion markers and instead uses visual cues to support non-linear iteration, and that the toolkit functions as an educational resource, a documentation tool, and a mechanism for continuous reflection.
Load-bearing premise
The toolkit's compliance value rests on the assumption that the academic team's internal audit faithfully captured every AI Act obligation for high-risk systems and translated each into a toolkit task, a mapping that is described but not externally validated.
Editorial extensions
If this is right
- Teams using the toolkit can generate a downloadable PDF record of their compliance work, usable for audits or internal review.
- Small and medium organizations, not only large firms, can access a free, open tool for navigating high-risk AI obligations under the AI Act.
- Design choices that avoid checkmarks and percentages signal that ethics work is ongoing, which may shift users' mental models away from one-time completion.
- The collaboration process, including value alignment, advisory board feedback, and user testing, can be adopted as a blueprint by other academic-industry pairs.
- The toolkit's Creative Commons license and templating system allow organizations to customize it for their own workflows while staying aligned with the Act.
Reading between the lines
- If the toolkit's audit of the AI Act is complete, it could serve as a low-cost baseline for compliance documentation, but that completeness is not independently verified, so prudent users would cross-check against official EU guidance.
- The 'spaces' structure could plausibly be adapted to other evolving regulations, such as the GDPR or sector-specific rules, where continuous reflection is also valuable.
- A natural next test is a longitudinal field study with a high-risk AI team using the toolkit across a full development cycle, comparing outcomes against teams using conventional checklists.
- The paper's claim about industry uptake would be strengthened by usage analytics from the open-access site, which the paper does not report.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports on a year-long cross-sectoral collaboration between a UK-based academic team and an Italy-based industry team to develop a free, open-access, web-based toolkit that guides practitioners through EU AI Act compliance while embedding a 'pro-justice' ethical orientation. It describes the collaboration process (value elicitation, advisory board feedback, user studies), the resulting toolkit structure (seven interconnected 'spaces' with tasks, rationales, and workbook documentation), and reflections on toolkit design and academia–industry partnership. The paper positions both the toolkit and the collaboration process as a blueprint for future responsible-AI tool development.
Significance. If the claims hold, the paper makes a constructive contribution to the AI ethics toolkit literature by providing a concrete, open-access resource and a detailed process account of how academic and industry partners can negotiate value commitments, regulatory requirements, and practical usability. The manuscript is honest about limitations such as evolving regulation, maintenance sustainability, and user uncertainty about compliance. The strengths include the public availability of the artifact, the two-round advisory board process with 23 experts, user studies with 74 participants, and the explicit discussion of tensions between linear and iterative design. However, the paper's central value claims—that the toolkit supports AI Act compliance and that it is usable and effective—rest on evidence that is currently either unreported or only anecdotally summarized.
major comments (3)
- [3.1.2, 3.2.3] The claim that the toolkit helps practitioners comply with the EU AI Act rests on the internal audit described in Section 3.1.2, where 'a member of the AT did a thorough audit of every topic covered by, and every task implied by, the EU AI Act.' However, the audit's method, its output (the obligation-to-task mapping), and its validation status are not reported. The statement in Section 3.2.3 that the advisory board feedback 'helped us ensure that the toolkit comprehensively addresses all obligations' is not a substitute for showing the mapping. Since a missed or mistranslated mandatory obligation would break the compliance claim, this is load-bearing evidence. Please report the audit methodology, present a table or appendix mapping each high-risk obligation (e.g., risk management, data governance, technical documentation, record-keeping, transparency, human oversight, accuracy/robustness/cybersecurity) to the corresponding toolkit spaces/tasks, and state the limits of the audit and how feedback addressed them.
- [3.3, 5.2.2] The user studies with 74 participants are summarized only anecdotally. For example, Section 5.2.2 states that 'User testing feedback indicated that the navigation of the toolkit was intuitive and comparable to product management applications,' but no participant counts, instruments, analysis procedures, or systematic results are given. This is load-bearing for the claim that the toolkit is a 'usable and meaningful' resource. Please provide a more systematic summary of the user studies (e.g., task success rates, rating scales, qualitative coding themes, participant breakdown across the three groups) or explicitly scale back the usability claims to what the reported evidence supports.
- [5.2.2] The paper honestly reports that participants asked, 'If I complete all the tasks within the toolkit, does that mean I'm fully compliant?' and that the authors addressed this with an onboarding expectations-management section. However, this response is not described, and the paper's abstract and introduction still assert that the toolkit helps practitioners 'comply with the AI Act.' Rather than treating this as a merely user-facing issue, the paper should clarify the intended status of the toolkit's outputs (e.g., documentation to prepare for audits, not a guarantee of full legal compliance) and align the stated claims with that status. This would strengthen the paper's honesty and avoid overclaiming.
minor comments (5)
- [2.1] The sentence 'we defined obligations set out by the EU through feminist ideas' is ambiguous; it likely means the authors interpreted the Act's obligations through feminist ideas. Please rephrase for clarity.
- [5.2.3] The phrase 'We approached this legally, through our choice of content license' is awkward; the choice of a Creative Commons license is a legal mechanism, but the sentence reads as if the license itself is the legal approach. Suggest rewording to clarify.
- [6] There is a typo in the final section: 'the otheress essential half of the equation' should be 'the other essential half of the equation.'
- [References] Several references have incomplete or inconsistent bibliographic details, including [24] and [29], which lack a work title in the rendered text. These should be completed for a camera-ready version.
- [6] The paper is framed as a 'blueprint' for other cross-sectoral teams, but the evidence is a single case study. The reflective lessons are valuable, yet the framing overstates the generalizability without comparative analysis or limitations on transferability. A more modest framing, such as 'a detailed case study with transferable insights,' would better match the evidence.
Circularity Check
No significant circularity: the paper is a reflective design case study rather than a derivation; its compliance claim rests on an unshown internal audit, which is an evidence gap, not a circular reduction.
full rationale
This is a process-and-design paper, not a formal derivation with equations, fitted parameters, or predictive claims that could reduce to their inputs by construction. The toolkit's pro-justice framing is grounded partly in the authors' own prior work, including a companion paper ([17]: 'The theoretical foundations for this pro-justice approach are summarised in the next section and elaborated in greater detail in a separate article') and the first author's earlier toolkit critiques ([15], [16]). These self-citations inform the design rationale, but they are not load-bearing in a derivation chain: no uniqueness theorem is imported, no alternative is ruled out by citation, and no result is made true by stipulation. The central claim that the toolkit 'comprehensively addresses all obligations outlined by the EU AI Act' (Section 3.2.3) is supported by an internal 'thorough audit of every topic covered by, and every task implied by, the EU AI Act' (Section 3.1.2) plus advisory feedback. That is a self-assessment rather than independent legal validation, and the paper itself records user uncertainty ('If I complete all the tasks within the toolkit, does that mean I'm fully compliant?', Section 5.2.2), answered with expectation management rather than proof of completeness. This is a missing-evidence concern, not circularity: the toolkit's compliance value is not defined as whatever the internal audit found, and no output is identical to an input by definition. Hence no circular step meets the evidentiary bar; the score reflects only the presence of minor, non-load-bearing self-citation in the framing.
Assumptions & free parameters
assumptions (3)
- domain assumption Pro-justice frameworks (feminist, anti-racist, disability justice) should be foundational to AI development and compliance.
- domain assumption The EU AI Act's obligations for high-risk systems can be exhaustively audited and translated into discrete, documentable tasks.
- domain assumption User feedback from short sessions with self-selected participants is a valid measure of toolkit usability, acceptability, and feasibility.
Cite this review
Pith. "Pith review of A Toolkit for Compliance, a Toolkit for Justice: Drawing on Cross-sectoral Expertise to Develop a Pro-justice EU AI Act Toolkit." pith.science (2026). https://pith.science/paper/KP5YPPKS
@misc{pith2026250517165,
author = {Pith},
title = {Pith review of: A Toolkit for Compliance, a Toolkit for Justice: Drawing on Cross-sectoral Expertise to Develop a Pro-justice EU AI Act Toolkit},
year = {2026},
howpublished = {\url{https://pith.science/paper/KP5YPPKS}},
note = {Machine review of arXiv:2505.17165}
}
read the original abstract
The introduction of the AI Act in the European Union presents the AI research and practice community with a set of new challenges related to compliance. While it is certain that AI practitioners will require additional guidance and tools to meet these requirements, previous research on toolkits that aim to translate the theory of AI ethics into development and deployment practice suggests that such resources suffer from multiple limitations. These limitations stem, in part, from the fact that the toolkits are either produced by industry-based teams or by academics whose work tends to be abstract and divorced from the realities of industry. In this paper, we discuss the challenge of developing an AI ethics toolkit for practitioners that helps them comply with new AI-focused regulation, but that also moves beyond mere compliance to consider broader socio-ethical questions throughout development and deployment. The toolkit was created through a cross-sectoral collaboration between an academic team based in the UK and an industry team in Italy. We outline the background and rationale for creating a pro-justice AI Act compliance toolkit, detail the process undertaken to develop it, and describe the collaboration and negotiation efforts that shaped its creation. We aim for the described process to serve as a blueprint for other teams navigating the challenges of academia-industry partnerships and aspiring to produce usable and meaningful AI ethics resources.
Figures
Reference graph
Works this paper leans on
-
[17]
In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI ’21)
Towards Fairness in Practice: A Practitioner- Oriented Rubric for Evaluating Fair ML Toolkits. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI ’21). Asso- ciation for Computing Machinery, New York, NY, USA, Article 236, 1–13. https://doi.org/10.1145/3411764.3445604. [26]Andrea Saltelli and Monica Di Fiore
arXiv 2021
-
[7]
[13]Daniel Greene, Anna Lauren Hoffmann, and Luke Stark
Top Trends in M&A for 2024 Report. [13]Daniel Greene, Anna Lauren Hoffmann, and Luke Stark
work page 2024
-
[8]
The Ethics of AI Ethics: An Evaluation of Guidelines. Minds & Machines 30, 99–120. https://doi.org/10.1007/s11023-020-09517-8. [15]Tomasz Hollanek
-
[9]
The ethico-politics of design toolkits: responsible AI tools, from big tech guidelines to feminist ideation cards. AI and Ethics. https: //doi.org/10.1007/s43681-024-00545-z. [16]Tomasz Hollanek and Maya Indira Ganesh
-
[10]
Blueprints, Toolkits, and Playbooks to Advance Diversity and Inclusion in AI
Easy Wins and Low Hanging Fruit. Blueprints, Toolkits, and Playbooks to Advance Diversity and Inclusion in AI. Institute of Network Cultures. https://doi.org/10.17863/CAM.113869. [17]Tomasz Hollanek, Yulu Pi, Dorian Peters, Selen Yakar, Eleanor Drage
-
[13]
From What to How: An Initial Review of Publicly Available AI Ethics Tools, Methods and Research to Translate Principles into Practices. Sci Eng Ethics 26, 2141–2168. https://doi.org/10.1007/s11948-019-00165-5. [22]Dorian Peters
-
[15]
In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23)
Playing with Power Tools: Design Toolkits and the Framing of Equity. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 392, 1–24. https://doi. org/10.1145/3544548.3581490 [24]Vivian Qiang, Jimin Rhim, and AJung Moon
arXiv 2023
-
[16]
https://doi.org/10.1007/s00146-023-01653-w
AI & Soc 39, 1975–1994. https://doi.org/10.1007/s00146-023-01653-w. [25]Brianna Richardson, Jean Garcia-Gathright, Samuel F. Way, Jennifer Thom, and Henriette Cramer
Show all 23 references
-
[19]
Ev- eryday Algorithm Auditing: Understanding the Power of Everyday Users in A Toolkit for Compliance, a Toolkit for Justice FAccT ’25, June 23–26, 2025, Athens, Greece Surfacing Harmful Algorithmic Behaviors. Proc. ACM Hum.-Comput. Interact. 5, CSCW2, Article 433 (October 2021...
2025 doi
-
[20]
In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI ’21)
The Landscape and Gaps in Open Source Fairness Toolkits. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 699, 1–13. https://doi.org/10.1145/3411764.3445261 [30]Ville Vakk...
2021
-
[21]
In Proceedings of the 26th In- ternational Conference on Evaluation and Assessment in Software Engineering (EASE ’22)
How Do Software Companies Deal with Artificial Intelligence Ethics? A Gap Analysis. In Proceedings of the 26th In- ternational Conference on Evaluation and Assessment in Software Engineering (EASE ’22). Association for Computing Machinery, New York, NY, USA, 100–109. https://d...
-
[23]
Seeing Like a Toolkit: How Toolkits Envision the Work of AI Ethics. Proc. ACM Hum.-Comput. Interact. 7, CSCW1, Article 145 (April 2023), 27 pages. https://doi.org/10.1145/ 3579621</bib>
2023
-
[33]
1038/450033a [19]Anna Jobin, Marcello Ienca, and Effy Vayena
https://doi.org/10. 1038/450033a [19]Anna Jobin, Marcello Ienca, and Effy Vayena. 2019.The global landscape of AI ethics guidelines. Nature Machine Intelligence. 1, 389–399. https://doi.org/10. 1038/s42256-019-0088-2. [20]Shannon Mattern
2019
-
[1984]
ACM27, 6 (1984), 556–563
An assessment of the prototyping approach to information systems development.Commun. ACM27, 6 (1984), 556–563. https://doi.org/10. 1145/358080.358095 [3]Ahmed Ansari
1984
-
[1988]
InHandbook of Human-Computer Interaction
Rapid Prototyping for User Interface Design. InHandbook of Human-Computer Interaction. Elsevier, 859–875. https: //doi.org/10.1016/B978-0-444-70536-5.50044-0 [32]Richmond Y. Wong, Michael A. Madaio, and Nick Merrill
-
[2016]
Nature 538, 311–313
There is a blind spot in AI research. Nature 538, 311–313. https://doi.org/10.1038/538311a [10]Catherine D’Ignazio and Lauren Klein. 2020 Data Feminism. MIT. [11]Jessica Fjeld, Nele Achten, Hannah Hilligoss, Adam Nagy, and Madhulika Sriku- mar
2020 doi
-
[2018]
Fairness and Abstraction in Sociotechnical Systems. 2019 ACM Conference on Fairness, Accountability, and Transparency (FAT*), 59-68, Available at SSRN: https://ssrn.com/abstract$=$3265913 [28]Hong Shen, Alicia DeVos, Motahhare Eslami, and Kenneth Holstein
2019
-
[2020]
Berkman Klein Center Research Publication No
Principled Artificial Intelligence: Mapping Consensus in Ethical and Rights-Based Approaches to Principles for AI (January 15, 2020). Berkman Klein Center Research Publication No. 2020-1. http://dx.doi.org/10.2139/ssrn.3518482.~ [12]Gartner
2020 doi
-
[2021]
Patterns (N Y)
Algorithmic injustice: a relational ethics approach. Patterns (N Y). 12, 2(2):100205. doi: 10.1016/j.patter.2021.100205. PMID: 33659914; PMCID: PMC7892355. [7]JudeBrowne, StephenCave, EleanorDrage, KerryMcInerney.2023.‘Introduction: Feminist AI’.. In: Feminist AI: Critical Per...
2021
-
[2022]
International Journal of Human–Computer Interaction, 0(0), 1–13
Wellbeing Supportive Design – Research-Based Guidelines for Supporting Psychological Wellbeing in User Experience. International Journal of Human–Computer Interaction, 0(0), 1–13. https://doi.org/10.1080/10447318. 2022.2089812 [23]Adrian Petterson, Keith Cheng, and Priyank Chandra
2022
-
[2023]
AI Ethics 3, 389–406
The ethics of AI business practices: a review of 47 AI ethics guidelines. AI Ethics 3, 389–406. https://doi.org/10.1007/s43681-022-00156-6. [5]JacquiAylingandAdrianeChapman.2022.PuttingAIethicstowork: arethetools fit for purpose?. AI Ethics 2, 405–429. https://doi.org/10.1007/...
2022 doi
-
[2024]
Big Data & Society, 11(1)
Tech workers’ perspec- tives on ethical issues in AI development: Foregrounding feminist approaches. Big Data & Society, 11(1). https://doi.org/10.1177/20539517231221780 [9]Kate Crawford and Ryan Calo
-
[2025]
arXiv:2504.20075
The EU AI Act in Development Practice: A Pro-justice Approach. arXiv:2504.20075. [18]Sheila Jasanoff
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.