Pith. sign in

REVIEW 2 major objections 5 minor 101 references

Agentic AI breaks the old foundations of security and privacy; the field needs new concepts, not just better patches.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 01:55 UTC pith:Z6UOUEKY

load-bearing objection Clean, timely expert synthesis of agentic-AI security/privacy open problems; useful agenda paper, not a new result, with the usual single-workshop completeness caveat. the 2 major comments →

arxiv 2607.06608 v1 pith:Z6UOUEKY submitted 2026-07-07 cs.CR cs.AIcs.HC

Security and Privacy in Agentic AI: Grand Challenges and Future Directions

Adam Jenkins , Agnieszka Kitkowska , Caterina Maidhof , Diego Paracuellos , Francesco Sovrano , Gonzalo Gabriel Mendez , Guillermo Suarez-Tangil , Hana Kopecka
show 17 more authors
Isabel Wagner Isabel Barbera Javier Carnerero-Cano Jide Edu Jose Luis Martin-Navarro Jose Such Josep Domingo-Ferrer Juan Carlos Carrillo Kopo Marvin Ramokapane Mark Cote Pablo Vellosillo Ramon Ruiz-Dolz Rongjun Ma Ruba Abu-Salma Sameer Patil William Seymour Xiao Zhan
This is my paper
classification cs.CR cs.AIcs.HC
keywords agentic AIsecurityprivacyaccountabilityconsenthuman oversightresiliencehorizon scanning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper reports a three-day horizon-scanning workshop with thirty experts from academia, industry, and government that mapped the security and privacy risks of AI systems that plan, coordinate, and act with reduced human oversight. Its central claim is that the concepts that currently organise the field—consent, contextual integrity, traceability, certification, and vulnerability—were built for bounded, deterministic, attributable systems, and agentic AI violates those assumptions. Across four themes the authors show how legal responsibility fragments across supply chains, how one-shot consent fails multi-step multi-agent workflows, how personalisation and influence create new forms of vulnerability, and how non-deterministic self-modifying behaviour defeats one-time audits and containment. A sympathetic reader cares because these systems are already moving from chatbots into proactive tools that hold user data, call external services, and chain actions; without reconceptualisation, risk will be hard-wired into everyday infrastructure before governance can catch up.

Core claim

The paper establishes that agentic AI requires reconceptualisation rather than refinement of the foundational security and privacy vocabulary. The workshop synthesised four interlocking themes—accountability and liability, consent and data context, human oversight and vulnerability, and transparency and failure recovery—and showed that each rests on assumptions of bounded, deterministic, attributable behaviour that agentic systems systematically break.

What carries the argument

Four-theme horizon-scanning map produced by a fixed-group affinity-mapping workshop: responsibility allocation across fragmented supply chains; multi-step multi-agent consent and memory; calibration of personalisation and emergent influence; continuous auditing and cascade containment for non-deterministic agents.

Load-bearing premise

That a single three-day workshop with thirty experts whose groups stayed fixed and whose notes were synthesised by facilitators produces a complete and unbiased set of grand challenges that can stand as the research agenda.

What would settle it

A subsequent independent horizon scan or large-scale empirical survey of agentic-AI incidents that systematically fails to reproduce the same four themes, or that finds the old consent-and-audit vocabulary still adequate once continuous monitoring is applied.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper reports a three-day horizon-scanning workshop (11–13 March 2026, UPV) that convened thirty experts from academia, industry, and government. Using fixed parallel groups, affinity mapping, and progressive scaffolding from use cases to barriers to interventions (Method §2, Fig. 1), the authors synthesize four grand-challenge themes for security and privacy of agentic AI: (1) responsibility, legal compliance, governance and liability; (2) consent, data and context; (3) human oversight, personalization and vulnerability; and (4) transparency, resilience and failure recovery. The central claim (Conclusion §7) is that foundational concepts—consent, contextual integrity, traceability, certification, vulnerability—were developed for bounded, deterministic, attributable systems and that agentic AI violates those assumptions, requiring reconceptualisation rather than incremental refinement. The manuscript is a qualitative agenda-setting piece that maps research opportunities under each theme and situates them against EU instruments (GDPR, AI Act) and recent technical literature.

Significance. If the synthesis is accepted as a credible expert-derived agenda, the paper supplies a timely, structured research map for a rapidly emerging subfield that has so far lacked systematic user-focused security and privacy treatment. The four-theme organisation, the explicit linkage of legal duties to technical mechanisms (Theme 1), the reframing of consent and contextual integrity for multi-step multi-agent workflows (Theme 2), and the treatment of cascade failures and continuous auditing (Theme 4) give concrete entry points for subsequent empirical and systems work. Strengths include transparent method description, multi-sector participation, and a clear statement that the contribution is qualitative horizon scanning rather than experimental validation. The work is therefore useful as a community reference even if individual research directions remain to be operationalised.

major comments (2)
  1. Method §2 and Fig. 1: the load-bearing claim that the four themes constitute the field’s grand challenges rests on a single three-day affinity-mapping exercise with fixed group membership and facilitator synthesis. No inter-rater reliability, external triangulation, or post-workshop validation is reported. The manuscript should either (a) add a short limitations subsection that bounds the completeness claim and describes how facilitator synthesis was checked against raw notes, or (b) supply a lightweight validation step (e.g., independent coding of a sample of notes or a short follow-up survey of participants). Without one of these, the central claim remains an expert synthesis rather than a demonstrated consensus agenda.
  2. Conclusion §7 and Theme 2 (“Contexts Beyond Contextual Integrity”): the assertion that foundational concepts “require reconceptualisation rather than refinement” is presented as the workshop’s synthesis, yet the text itself leaves open whether contextual integrity should be revised, supplemented or replaced. The manuscript should state more precisely which concepts are claimed to be irreparably violated versus which can be extended, and should flag this as a contested workshop outcome rather than a settled result. A short table or paragraph mapping each foundational concept to the specific agentic-AI property that breaks it would make the claim falsifiable and easier for subsequent work to test.
minor comments (5)
  1. Throughout: several typographical and orthographic inconsistencies appear (e.g., “organisztional”, “localisztion”, “adversatial”, “infuence”, “becuase”, “na ¨ıve”). A careful copy-edit pass is needed.
  2. §1 and Method: OpenClaw is used as the running illustrative example; a one-sentence technical description of its capabilities would help readers unfamiliar with the system.
  3. Theme 1: the discussion of XAI as “nascent support infrastructure” is useful but could briefly distinguish source-faithful versus post-hoc explanations when linking them to liability assessment.
  4. References: a few arXiv/preprint entries lack final venue or DOI information; standardise where possible.
  5. Fig. 1 caption: the figure is described as illustrating “the four main general themes that emerged”; ensure the figure itself labels the four themes consistently with the section headings.

Circularity Check

0 steps flagged

No significant circularity: qualitative workshop synthesis with no fitted predictions, self-definitional claims, or load-bearing self-citation chains.

full rationale

This paper reports outcomes of a three-day horizon-scanning workshop (Method §2, Fig. 1) that used affinity mapping to organize expert discussion into four themes and research directions. It advances no quantitative derivation, no fitted parameters renamed as predictions, no uniqueness theorem, and no first-principles result that reduces to its inputs by construction. The central claim in §7—that foundational security/privacy concepts (consent, contextual integrity, traceability, certification, vulnerability) were developed for bounded, deterministic, attributable systems and that agentic AI requires reconceptualisation—is presented as a qualitative synthesis of workshop outputs, not as a forced mathematical or empirical prediction. Self-citations (e.g., Sovrano on XAI/compliance, Such/Zhan/Ma on privacy and agentic risks) appear as ordinary background literature supporting individual points within themes; none is load-bearing for the four-theme structure or the reconceptualisation claim by construction. The paper is self-contained as a transparent qualitative agenda-setting exercise; no circular step of the enumerated kinds is present.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 2 invented entities

As a qualitative horizon-scan report the paper introduces no free parameters or fitted constants. Its load-bearing premises are domain assumptions about the nature of agentic AI and the adequacy of expert elicitation; the four themes function as organizing constructs invented for the paper rather than independently measured entities.

axioms (3)
  • domain assumption Agentic AI systems plan, coordinate, and execute multi-step workflows with reduced human oversight and therefore differ qualitatively from passive chatbots.
    Stated in Introduction and used throughout to justify why existing consent and accountability models fail.
  • domain assumption A three-day affinity-mapping workshop with thirty international experts is a valid method for identifying grand challenges.
    Method §2; the entire claim structure rests on the outputs of this process.
  • domain assumption Existing regulatory instruments (GDPR, AI Act, Product Liability Directive) allocate duties to roles that do not map cleanly onto end-to-end agentic workflows.
    Theme 1; used to motivate the responsibility-allocation research opportunity.
invented entities (2)
  • Four grand-challenge themes (responsibility/liability; consent/data/context; human oversight/personalization/vulnerability; transparency/resilience/failure recovery) no independent evidence
    purpose: Organize workshop outputs into a research agenda
    Emergent clustering produced by facilitators and LLM-assisted summarization; no independent measurement outside the workshop.
  • Agentic AI context no independent evidence
    purpose: Name the conceptual gap left by classical contextual integrity
    Introduced in Theme 2 as a label for needed new frameworks; no external operational definition supplied.

pith-pipeline@v1.1.0-grok45 · 24704 in / 2260 out tokens · 28134 ms · 2026-07-11T01:55:53.268343+00:00 · methodology

0 comments
read the original abstract

We present key challenges and future research directions in the security and privacy of agentic AI, based on a horizon-scanning exercise that brought together thirty leading international experts from academia, industry, and government to engage in focused discussions and collaborative exercises on the emerging risks associated with the growing agency of AI.

Figures

Figures reproduced from arXiv: 2607.06608 by Adam Jenkins, Agnieszka Kitkowska, Caterina Maidhof, Diego Paracuellos, Francesco Sovrano, Gonzalo Gabriel Mendez, Guillermo Suarez-Tangil, Hana Kopecka, Isabel Barbera, Isabel Wagner, Javier Carnerero-Cano, Jide Edu, Jose Luis Martin-Navarro, Josep Domingo-Ferrer, Jose Such, Juan Carlos Carrillo, Kopo Marvin Ramokapane, Mark Cote, Pablo Vellosillo, Ramon Ruiz-Dolz, Rongjun Ma, Ruba Abu-Salma, Sameer Patil, William Seymour, Xiao Zhan.

Figure 1
Figure 1. Figure 1: Horizon-scanning workshop procedure and the output reported in this work [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

101 extracted references · 101 canonical work pages · 1 internal anchor

  1. [1]

    Introducing chatgpt,

    OpenAI, “Introducing chatgpt,” Nov. 2022, accessed: 2026-05-12. [Online]. Available: https://openai.com/index/chatgpt/ 10

    work page 2022

  2. [2]

    Privacy perceptions of custom gpts by users and creators,

    R. Ma, C. Maidhof, J. C. Carrillo, J. Lindqvist, and J. Such, “Privacy perceptions of custom gpts by users and creators,” inProceedings of the 2025 CHI Conference on Human Factors in Computing Systems, 2025, pp. 1–18

    work page 2025

  3. [3]

    Openclaw — personal ai assistant,

    OpenClaw, “Openclaw — personal ai assistant,” 2026, publication date unavailable; accessed: 2026-05-12. [Online]. Available: https://openclaw.ai/

    work page 2026

  4. [4]

    Formalizing and benchmarking prompt injection attacks and defenses,

    Y. Liu, Y. Jia, R. Geng, J. Jia, and N. Z. Gong, “Formalizing and benchmarking prompt injection attacks and defenses,” in 33rd USENIX Security Symposium (USENIX Security 24), 2024, pp. 1831–1847. [Online]. Available: https://www.usenix.org/ conference/usenixsecurity24/presentation/liu-yupei

    work page 2024

  5. [5]

    Gptracker: A large- scale measurement of misused gpts,

    X. Shen, Y. Shen, M. Backes, and Y. Zhang, “Gptracker: A large- scale measurement of misused gpts,” in2025 IEEE Symposium on Security and Privacy (SP). IEEE, 2025, pp. 336–354

    work page 2025

  6. [6]

    Malicious {LLM-Based}conversational{AI}makes users reveal personal information,

    X. Zhan, J. C. Carrillo, W. Seymour, and J. Such, “Malicious {LLM-Based}conversational{AI}makes users reveal personal information,” in34th USENIX Security Symposium (USENIX Security 25), 2025, pp. 61–80. [Online]. Available: https: //www.usenix.org/system/files/usenixsecurity25-zhan.pdf

    work page 2025

  7. [7]

    On concepts and methods in horizon scanning: Lessons from initiating policy dialogues on emerging issues,

    E. Amanatidou, M. Butter, V . Carabias, T. K ¨onn¨ol¨a, M. Leis, O. Sar- itas, P . Schaper-Rinkel, and V . Van Rij, “On concepts and methods in horizon scanning: Lessons from initiating policy dialogues on emerging issues,”Science and Public Policy, vol. 39, no. 2, pp. 208– 221, 2012

    work page 2012

  8. [8]

    Using affinity diagrams to evaluate interactive proto- types,

    A. Lucero, “Using affinity diagrams to evaluate interactive proto- types,” inIFIP conference on human-computer interaction. Springer, 2015, pp. 231–248

    work page 2015

  9. [9]

    Regulatory challenges of robotics: some guidelines for addressing legal and ethical issues,

    R. Leenes, E. Palmerini, B.-J. Koops, A. Bertolini, P . Salvini, and F. Lucivero, “Regulatory challenges of robotics: some guidelines for addressing legal and ethical issues,”Law, Innovation and Tech- nology, vol. 9, no. 1, pp. 1–44, 2017

    work page 2017

  10. [10]

    Find the gap: Ai, responsible agency and vulnerability,

    S. Vallor and T. Vierkant, “Find the gap: Ai, responsible agency and vulnerability,”Minds and Machines, vol. 34, no. 3, p. 20, 2024

    work page 2024

  11. [11]

    Harms from increasingly agentic algorithmic systems,

    A. Chan, R. Salganik, A. Markelius, C. Pang, N. Rajkumar, D. Krasheninnikov, L. Langosco, Z. He, Y. Duan, M. Carroll et al., “Harms from increasingly agentic algorithmic systems,” in Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, 2023, pp. 651–666

    work page 2023

  12. [12]

    General data protection regulation,

    European Union, “General data protection regulation,” Regulation (EU) 2016/679, 2016. [Online]. Available: https://eur-lex.europa. eu/eli/reg/2016/679/oj

    work page 2016

  13. [13]

    Artificial Intelligence Act,

    ——, “Artificial Intelligence Act,” Regulation (EU) 2024/1689,

    work page 2024

  14. [14]

    Available: https://eur-lex.europa.eu/eli/reg/ 2024/1689/oj

    [Online]. Available: https://eur-lex.europa.eu/eli/reg/ 2024/1689/oj

    work page 2024

  15. [15]

    Charter of fundamental rights of the european union,

    ——, “Charter of fundamental rights of the european union,” 2012/C 326/02, 2012. [Online]. Available: https://eur-lex.europa. eu/eli/treaty/char 2012/oj

    work page 2012

  16. [16]

    An objective metric for explainable AI: How and why to estimate the degree of explainability,

    F. Sovrano and F. Vitali, “An objective metric for explainable AI: How and why to estimate the degree of explainability,”Knowledge- Based Systems, vol. 278, p. 110866, 2023

    work page 2023

  17. [17]

    Legal XAI: A systematic review and interdisciplinary mapping of XAI and EU law, towards a research agenda for legally responsible AI,

    F. Sovrano, G. Vilone, M. Lognoul, and L. Longo, “Legal XAI: A systematic review and interdisciplinary mapping of XAI and EU law, towards a research agenda for legally responsible AI,” SSRN preprint, 2026, under review

    work page 2026

  18. [18]

    Understanding accountability in algorithmic supply chains,

    J. Cobbe, M. Veale, and J. Singh, “Understanding accountability in algorithmic supply chains,” inProceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, 2023, pp. 1186–1197

    work page 2023

  19. [19]

    Ai agents under eu law,

    L. Nannini, A. L. Smith, M. J. Maggini, E. Panai, S. Feliciano, A. Tiulkanov, E. Maran, J. Gealy, and P . Bisconti, “Ai agents under eu law,” 2026

    work page 2026

  20. [20]

    Digital services act,

    European Union, “Digital services act,” Regulation (EU) 2022/2065, 2022. [Online]. Available: https://eur-lex.europa.eu/ eli/reg/2022/2065/oj

    work page 2022

  21. [21]

    Medical device regulation,

    ——, “Medical device regulation,” Regulation (EU) 2017/745,

    work page 2017

  22. [22]

    Available: https://eur-lex.europa.eu/eli/reg/ 2017/745/oj

    [Online]. Available: https://eur-lex.europa.eu/eli/reg/ 2017/745/oj

    work page 2017

  23. [23]

    Product liability directive,

    ——, “Product liability directive,” Directive (EU) 2024/2853,

    work page 2024

  24. [24]

    Available: https://eur-lex.europa.eu/eli/dir/ 2024/2853/oj

    [Online]. Available: https://eur-lex.europa.eu/eli/dir/ 2024/2853/oj

    work page 2024

  25. [25]

    On the quest for effectiveness in human oversight: Interdisciplinary perspectives,

    S. Sterz, K. Baum, S. Biewer, H. Hermanns, A. Lauber-R ¨onsberg, P . Meinel, and M. Langer, “On the quest for effectiveness in human oversight: Interdisciplinary perspectives,” inProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, 2024

    work page 2024

  26. [26]

    Fostering appropriate reliance on large language models: The role of explanations, sources, and inconsistencies,

    S. S. Y. Kim, J. W. Vaughan, Q. V . Liao, T. Lombrozo, and O. Russakovsky, “Fostering appropriate reliance on large language models: The role of explanations, sources, and inconsistencies,” inProceedings of the 2025 CHI Conference on Human Factors in Computing Systems, ser. CHI ’25. New York, NY, USA: Association for Computing Machinery, 2025. [Online]. A...

    work page doi:10.1145/3706598.3714020 2025

  27. [27]

    Software documentation: The practitioners’ perspective,

    E. Aghajani, C. Nagy, M. Linares-V ´asquez, L. Moreno, G. Bavota, M. Lanza, and D. C. Shepherd, “Software documentation: The practitioners’ perspective,” inProceedings of the ACM/IEEE 42nd In- ternational Conference on Software Engineering, ser. ICSE ’20. ACM, 2020, pp. 590–601

    work page 2020

  28. [28]

    Software documentation issues unveiled,

    E. Aghajani, C. Nagy, O. L. Vega-M ´arquez, M. Linares-V ´asquez, L. Moreno, G. Bavota, and M. Lanza, “Software documentation issues unveiled,” inProceedings of the 41st International Conference on Software Engineering, ser. ICSE ’19, 2019, pp. 1199–1210

    work page 2019

  29. [29]

    Detecting outdated code el- ement references in software repository documentation,

    W. S. Tan, M. Wagner, and C. Treude, “Detecting outdated code el- ement references in software repository documentation,”Empirical Software Engineering, vol. 29, no. 5, 2024

    work page 2024

  30. [30]

    An empirical study on compliance with ranking transparency in the software documenta- tion of EU online platforms,

    F. Sovrano, M. Lognoul, and A. Bacchelli, “An empirical study on compliance with ranking transparency in the software documenta- tion of EU online platforms,” inProceedings of the 46th International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS 2024). Lisbon, Portugal: Association for Computing Machinery, 2024, pp. 46–56

    work page 2024

  31. [31]

    Simplifying software compliance: AI technologies in drafting technical docu- mentation for the AI Act,

    F. Sovrano, E. Hine, S. Anzolut, and A. Bacchelli, “Simplifying software compliance: AI technologies in drafting technical docu- mentation for the AI Act,”Empirical Software Engineering, vol. 30, no. 4, p. 91, 2025

    work page 2025

  32. [32]

    Outlining traceability: A principle for operationalizing accountability in computing systems,

    J. A. Kroll, “Outlining traceability: A principle for operationalizing accountability in computing systems,” inProceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, ser. FAccT ’21. ACM, 2021

    work page 2021

  33. [33]

    Closing the ai accountability gap: Defining an end-to-end framework for internal algorithmic auditing,

    I. D. Raji, A. Smart, R. N. White, M. Mitchell, T. Gebru, B. Hutchin- son, J. Smith-Loud, D. Theron, and P . Barnes, “Closing the ai accountability gap: Defining an end-to-end framework for internal algorithmic auditing,” inProceedings of the 2020 Conference on Fairness, Accountability, and Transparency, ser. FAT* ’20. ACM, 2020

    work page 2020

  34. [34]

    Audit trails for accountability in large language models,

    V . Ojewale, H. Suresh, and S. Venkatasubramanian, “Audit trails for accountability in large language models,” 2026

    work page 2026

  35. [35]

    Regulation (eu) 2024/1689 laying down harmonised rules on artificial intelligence,

    European Parliament and Council of the European Union, “Regulation (eu) 2024/1689 laying down harmonised rules on artificial intelligence,” 2024, artificial Intelligence Act, especially Article 12 on record-keeping. [Online]. Available: https://eur-lex.europa.eu/eli/reg/2024/1689/oj

    work page 2024

  36. [36]

    Accountability capture: How record-keeping to support ai trans- parency and accountability (re)shapes algorithmic oversight,

    S. Chappidi, J. Cobbe, C. Norval, A. Mazumder, and J. Singh, “Accountability capture: How record-keeping to support ai trans- parency and accountability (re)shapes algorithmic oversight,” 2025

    work page 2025

  37. [37]

    Regulation (eu) 2016/679: General data protection regulation,

    European Parliament and Council of the European Union, “Regulation (eu) 2016/679: General data protection regulation,” 2016, especially Article 5 on purpose limitation, data minimisation, storage limitation, and accountability. [Online]. Available: https://eur-lex.europa.eu/eli/reg/2016/679/oj

    work page 2016

  38. [38]

    Regulation (eu) 2016/679: Article 32, security of pro- cessing,

    ——, “Regulation (eu) 2016/679: Article 32, security of pro- cessing,” 2016, technical and organisational measures, including confidentiality, integrity, pseudonymisation, and encryption. [On- line]. Available: https://eur-lex.europa.eu/eli/reg/2016/679/oj

    work page 2016

  39. [39]

    Harpocrates: Privacy-preserving and immutable audit log for sensitive data operations,

    M. B. Thazhath, J. Michalak, and T. Hoang, “Harpocrates: Privacy-preserving and immutable audit log for sensitive data operations,” in2022 IEEE 4th International Conference on Trust, Privacy and Security in Intelligent Systems, and Applications, ser. TPS-ISA. IEEE, 2022. [Online]. Available: https://ieeexplore.ieee.org/document/10063383

    work page arXiv 2022

  40. [40]

    On cryptographic mechanisms for the selective dis- closure of verifiable credentials,

    A. Flamini, G. Sciarretta, M. Scuro, A. Sharif, A. Tomasi, and S. Ranise, “On cryptographic mechanisms for the selective dis- closure of verifiable credentials,” 2024

    work page 2024

  41. [41]

    Role-based access controls,

    D. F. Ferraiolo and D. R. Kuhn, “Role-based access controls,” inProceedings of the 15th National Computer Security Conference,

    work page

  42. [42]

    Available: https://csrc.nist.gov/pubs/conference/ 1992/10/13/rolebased-access-controls/final

    [Online]. Available: https://csrc.nist.gov/pubs/conference/ 1992/10/13/rolebased-access-controls/final

    work page 1992

  43. [43]

    A framework for understanding sources of harm throughout the machine learning life cycle,

    H. Suresh and J. V . Guttag, “A framework for understanding sources of harm throughout the machine learning life cycle,” inProceedings of the 1st ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, ser. EAAMO ’21. ACM, 2021

    work page 2021

  44. [44]

    Cognitive bias in decision-making with llms,

    J. M. Echterhoff, Y. Liu, A. Alessa, J. McAuley, and Z. He, “Cognitive bias in decision-making with llms,” inFindings of the 11 Association for Computational Linguistics: EMNLP 2024. Association for Computational Linguistics, 2024, pp. 12 640–12 653

    work page 2024

  45. [45]

    Towards understanding sycophancy in language models,

    M. Sharma, M. Tong, T. Korbak, D. Duvenaud, A. Askell, S. R. Bowman, N. Cheng, E. Durmus, Z. Hatfield-Dodds, S. R. Johnston, S. Kravec, T. Maxwell, S. McCandlish, K. Ndousse, O. Rausch, N. Schiefer, D. Yan, M. Zhang, and E. Perez, “Towards understanding sycophancy in language models,” inInternational Conference on Learning Representations, 2024. [Online]....

    work page 2024

  46. [46]

    Agentharm: A bench- mark for measuring harmfulness of llm agents,

    M. Andriushchenko, A. Souly, M. Dziemian, D. Duenas, M. Lin, J. Wang, D. Hendrycks, A. Zou, Z. Kolter, M. Fredrikson, E. Winsor, J. Wynne, Y. Gal, and X. Davies, “Agentharm: A bench- mark for measuring harmfulness of llm agents,” inInternational Conference on Learning Representations, 2025. [Online]. Avail- able: https://proceedings.iclr.cc/paper files/pa...

    work page 2025

  47. [47]

    Mitigating prompt- induced cognitive biases in general-purpose ai for software engi- neering,

    F. Sovrano, G. Dominici, and A. Bacchelli, “Mitigating prompt- induced cognitive biases in general-purpose ai for software engi- neering,” inProceedings of the ACM International Conference on the Foundations of Software Engineering, 2026

    work page 2026

  48. [48]

    Is general-purpose ai reasoning sensitive to data- induced cognitive biases? dynamic benchmarking on typical soft- ware engineering dilemmas,

    F. Sovrano, G. Dominici, R. Sevastjanova, A. Stramiglio, and A. Bacchelli, “Is general-purpose ai reasoning sensitive to data- induced cognitive biases? dynamic benchmarking on typical soft- ware engineering dilemmas,”ACM Transactions on Software Engi- neering and Methodology, 2026

    work page 2026

  49. [49]

    Large language models for in-file vulnerability localization can be “lost in the end

    F. Sovrano, A. Bauer, and A. Bacchelli, “Large language models for in-file vulnerability localization can be “lost in the end”,” in Proceedings of the ACM International Conference on the Foundations of Software Engineering, 2025

    work page 2025

  50. [50]

    Shostack,Threat Modeling: Designing for Security

    A. Shostack,Threat Modeling: Designing for Security. Wiley, 2014. [Online]. Available: https://shostack.org/books/ threat-modeling-book

    work page 2014

  51. [51]

    Artificial intelligence risk management framework (AI RMF 1.0),

    E. Tabassi, “Artificial intelligence risk management framework (AI RMF 1.0),” National Institute of Standards and Technology, Gaithersburg, MD, Tech. Rep. NIST AI 100-1, 2023

    work page 2023

  52. [52]

    STPA handbook,

    N. G. Leveson and J. P . Thomas, “STPA handbook,” MIT Partnership for Systems Approaches to Safety, Tech. Rep. MIT-STAMP-001, 2018. [Online]. Available: https://psas.scripts. mit.edu/home/get file.php?name=STPA handbook.pdf

    work page 2018

  53. [53]

    Huang and C

    K. Huang and C. Hughes,Agentic AI Threat Modeling, ser. Ad- vances in Data Analytics, AI, and Smart Systems. Cham: Springer, 2025, pp. 17–50

    work page 2025

  54. [54]

    Multi-agent risks from advanced AI,

    L. Hammond, A. Chan, J. Clifton, J. Hoelscher-Obermaier, A. Khan, E. McLean, C. Smith, W. Barfuss, J. Foerster, T. Gavenˇciak, T. A. Han, E. Hughes, V . Kova ˇr´ık, J. Kulveit, J. Z. Leibo, C. Oesterheld, C. Schroeder de Witt, N. Shah, M. Wellman, P . Bova, T. Cimpeanu, C. Ezell, Q. Feuillade-Montixi, M. Franklin, E. Kran, I. Krawczuk, M. Lamparth, N. Lau...

    work page 2025

  55. [55]

    Why do multi-agent LLM systems fail?

    M. Cemri, M. Z. Pan, S. Yang, L. A. Agrawal, B. Chopra, R. Tiwari, K. Keutzer, A. Parameswaran, D. Klein, K. Ramchandran, M. A. Zaharia, J. E. Gonzalez, and I. Stoica, “Why do multi-agent LLM systems fail?” inAdvances in Neural Information Processing Systems, vol. 38, 2025, datasets and Benchmarks Track. [Online]. Available: https://proceedings.neurips.cc...

    work page 2025

  56. [56]

    m&m’s: A benchmark to evaluate tool-use for multi-step multi-modal tasks,

    Z. Ma, W. Huang, J. Zhang, T. Gupta, and R. Krishna, “m&m’s: A benchmark to evaluate tool-use for multi-step multi-modal tasks,” inComputer Vision – ECCV 2024, ser. Lecture Notes in Computer Science, vol. 15068. Cham: Springer, 2025, pp. 18–34

    work page 2024

  57. [57]

    Are more LLM calls all you need? towards the scaling properties of compound AI systems,

    L. Chen, J. Davis, B. Hanin, P . Bailis, I. Stoica, M. Zaharia, and J. Zou, “Are more LLM calls all you need? towards the scaling properties of compound AI systems,” inAdvances in Neural Infor- mation Processing Systems, vol. 37, 2024

    work page 2024

  58. [58]

    Facilitating threat modeling by leveraging large language models,

    I. Elsharef, Z. Zeng, and Z. Gu, “Facilitating threat modeling by leveraging large language models,” inProceedings of the NDSS Symposium Workshop on AI Systems with Confidential Computing,

    work page

  59. [59]

    Available: https://www.ndss-symposium.org/ wp-content/uploads/aiscc2024-16-paper.pdf

    [Online]. Available: https://www.ndss-symposium.org/ wp-content/uploads/aiscc2024-16-paper.pdf

    work page

  60. [60]

    Measuring progress on scalable oversight for large language models,

    S. R. Bowman, J. Hyun, E. Perez, E. Chen, C. Pettit, S. Heiner et al., “Measuring progress on scalable oversight for large language models,” 2022

    work page 2022

  61. [61]

    Dignum,Responsible Artificial Intelligence: How to Develop and Use AI in a Responsible Way

    V . Dignum,Responsible Artificial Intelligence: How to Develop and Use AI in a Responsible Way. Cham: Springer, 2019

    work page 2019

  62. [62]

    What to account for when accounting for algo- rithms: A systematic literature review on algorithmic accountabil- ity,

    M. Wieringa, “What to account for when accounting for algo- rithms: A systematic literature review on algorithmic accountabil- ity,” inProceedings of the 2020 Conference on Fairness, Accountability, and Transparency, ser. FAT* ’20. ACM, 2020, pp. 1–18

    work page 2020

  63. [63]

    Accountability in multi-agent organizations: From conceptual design to agent programming,

    M. Baldoni, C. Baroglio, O. Boissier, K. M. May, R. Micalizio, and S. Tedeschi, “Accountability in multi-agent organizations: From conceptual design to agent programming,”Autonomous Agents and Multi-Agent Systems, vol. 37, no. 1, p. 7, 2023

    work page 2023

  64. [64]

    AI safety via debate,

    G. Irving, P . Christiano, and D. Amodei, “AI safety via debate,” 2018

    work page 2018

  65. [65]

    Improving factuality and reasoning in language models through multiagent debate,

    Y. Du, S. Li, A. Torralba, J. B. Tenenbaum, and I. Mordatch, “Improving factuality and reasoning in language models through multiagent debate,” inProceedings of the 41st International Confer- ence on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 235. PMLR, 2024, pp. 11 733–11 763

    work page 2024

  66. [66]

    Value sensitive design and information systems,

    B. Friedman, P . H. J. Kahn, and A. Borning, “Value sensitive design and information systems,” inEarly Engagement and New Technolo- gies: Opening Up the Laboratory, ser. Philosophy of Engineering and Technology, N. Doorn, D. Schuurbiers, I. van de Poel, and M. E. Gorman, Eds. Dordrecht: Springer, 2013, vol. 16, pp. 55–95

    work page 2013

  67. [67]

    Fairness and abstraction in sociotechnical systems,

    A. D. Selbst, D. Boyd, S. A. Friedler, S. Venkatasubramanian, and J. Vertesi, “Fairness and abstraction in sociotechnical systems,” in Proceedings of the Conference on Fairness, Accountability, and Trans- parency, ser. FAT* ’19. ACM, 2019, pp. 59–68

    work page 2019

  68. [68]

    Inherent trade- offs in the fair determination of risk scores,

    J. Kleinberg, S. Mullainathan, and M. Raghavan, “Inherent trade- offs in the fair determination of risk scores,” in8th Innovations in Theoretical Computer Science Conference, ser. Leibniz International Proceedings in Informatics, vol. 67. Schloss Dagstuhl–Leibniz- Zentrum f”ur Informatik, 2017, pp. 43:1–43:23

    work page 2017

  69. [69]

    Big data’s disparate impact,

    S. Barocas and A. D. Selbst, “Big data’s disparate impact,”Califor- nia Law Review, vol. 104, pp. 671–732, 2016

    work page 2016

  70. [70]

    The (im) possibility of fairness: Different value systems require differ- ent mechanisms for fair decision making,

    S. A. Friedler, C. Scheidegger, and S. Venkatasubramanian, “The (im) possibility of fairness: Different value systems require differ- ent mechanisms for fair decision making,”Communications of the ACM, vol. 64, no. 4, pp. 136–143, 2021

    work page 2021

  71. [71]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P . Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P . Welinder, P . Christiano, J. Leike, and R. Lowe, “Training language models to follow instructions with human feedback,” inAdvances in Neural Information Processing Systems, v...

    work page 2022

  72. [72]

    Constitutional AI: Harmlessness from AI feedback,

    Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, C. Chen, C. Olsson, C. Olah, D. Hernandez, D. Drain, D. Ganguli, D. Li, E. Tran-Johnson, E. Perez, J. Kerr, J. Mueller, J. Ladish, J. Lan- dau, K. Ndousse, K. Lukosuite, L. Lovitt, M. Sellitto, N. Elhage, N. Schiefer, N. Mercado, N. DasSarma, R....

    work page 2022

  73. [73]

    Jailbroken: How does LLM safety training fail?

    A. Wei, N. Haghtalab, and J. Steinhardt, “Jailbroken: How does LLM safety training fail?” inAdvances in Neural Information Pro- cessing Systems, vol. 36, 2023

    work page 2023

  74. [74]

    Universal and transferable adversarial attacks on aligned language models,

    A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrik- son, “Universal and transferable adversarial attacks on aligned language models,” 2023

    work page 2023

  75. [75]

    Small Language Models are the Future of Agentic AI

    P . Belcak, G. Heinrich, S. Diao, Y. Fu, X. Dong, S. Muralidharan, Y. C. Lin, and P . Molchanov, “Small language models are the future of agentic ai,”arXiv preprint arXiv:2506.02153, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  76. [76]

    Can global XAI methods reveal injected behaviours in LLMs? SHAP vs. rule extraction vs. RuleSHAP,

    F. Sovrano, “Can global XAI methods reveal injected behaviours in LLMs? SHAP vs. rule extraction vs. RuleSHAP,” inProceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V .2 (KDD ’26). Jeju Island, Republic of Korea: Association for Computing Machinery, 2026

    work page 2026

  77. [77]

    Neuron-anchored rule extraction for large language models via contrastive hierarchi- cal ablation,

    F. Sovrano, G. Dominici, and M. Langheinrich, “Neuron-anchored rule extraction for large language models via contrastive hierarchi- cal ablation,” inProceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining, ser. KDD ’26. Jeju Island, Republic of Korea: Association for Computing Machinery, 2026

    work page 2026

  78. [78]

    Illocutionary explanation planning for source-faithful explanations in retrieval-augmented language models,

    F. Sovrano and A. Bacchelli, “Illocutionary explanation planning for source-faithful explanations in retrieval-augmented language models,” inProceedings of the 4th World Conference on eXplainable 12 Artificial Intelligence (xAI 2026), ser. Communications in Computer and Information Science. Springer, 2026

    work page 2026

  79. [79]

    Assessing model-agnostic XAI methods against EU AI Act explainability requirements,

    F. Sovrano, G. Vilone, and M. Lognoul, “Assessing model-agnostic XAI methods against EU AI Act explainability requirements,” inProceedings of the 4th World Conference on eXplainable Artificial Intelligence (xAI 2026), ser. Communications in Computer and Information Science. Springer, 2026

    work page 2026

  80. [80]

    AI deception: A survey of examples, risks, and potential solu- tions,

    P . S. Park, S. Goldstein, A. O’Gara, M. Chen, and D. Hendrycks, “AI deception: A survey of examples, risks, and potential solu- tions,”Patterns, vol. 5, no. 5, p. 100988, 2024

    work page 2024

Showing first 80 references.