Effective Explanations for Belief-Desire-Intention Robots: When and What to Explain

Cong Wang; Roberto Calandra; Verena Kl\"os

REVIEW 5 major objections 5 minor 23 references

Effective Explanations for Belief-Desire-Intention Robots: When and What to Explain

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A robot should explain when its action surprises the user, and the explanation should say the intention and the context, not just the action or the goal.

desk verdict A useful but modest user survey plus a pair of BDI explanation algorithms; the empirical core is decent, the algorithm validation is missing, and the surprise trigger is an untested proxy. read the letter →

arxiv 2507.02016 v1 pith:NQ6TKL7C submitted 2025-07-02 cs.RO cs.AI

Cong Wang , Roberto Calandra , Verena Klös This is my paper

classification cs.ROcs.AI

keywords explainableroboticsBDIagentssurprise-triggeredexplanationcontentuserstudyhuman-robotinteractionhierarchicalintentioncontextAgentSpeak

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish when a home-assistant robot should volunteer an explanation and what that explanation should contain. From an online scenario-based survey of 33 participants, it concludes that the stronger a user's surprise at a robot action, the more likely they want an explanation, and that the most useful explanations are concise statements of the intention behind the confusing action together with the contextual factors that made the robot choose it. Isolated action descriptions and high-level goal statements score much lower. The authors then turn these findings into two algorithms for Belief-Desire-Intention (BDI) robots: one that flags an action as surprising when it falls outside a learned set of expected successors, and one that assembles an explanation from the hierarchical chain of intention contexts. If correct, the work gives BDI robots a lightweight, reasoner-friendly way to decide when and what to explain during ongoing tasks.

What carries the argument

The mechanism is a pair of lightweight algorithms that sit on top of the standard BDI reasoning cycle. Algorithm 1 takes the user's expectation to be a function $N: A \to 2^A$ mapping each action to its set of expected successors; an action $a_{i+1}$ is flagged as surprising whenever $a_{i+1} \notin N(a_i)$, which triggers an explanation and then adds $a_{i+1}$ to $N(a_i)$ so the behavior becomes routine. Algorithm 2 constructs the explanation from the plan hierarchy: it forms $\Gamma_i = \bigcup_{\pi \in P(\pi_i)} \gamma(\pi)$, the union of the context conditions of the current intention and its parent intentions, and returns the suffix $(a_i, \ldots, a_k)$ where $k$ is the first index whose action's effects intersect $\Gamma_i$. Together they convert the survey finding 'explain the key contextual factors' into a syntactic operation on AgentSpeak plan structures.

What would settle it

Present participants with a BDI robot's plan and, for the same pair of consecutive actions, vary only the cost or rationality of the transition (e.g., opening the dishwasher first because the robot is already holding cups versus doing so without any such reason), and compare their surprise ratings with the predictions of $N(a_i)$; if people are surprised by transitions inside $N$ or unsurprised by transitions outside $N$, the successor-set trigger is not tracking the construct the survey measured.

Watch

Extended reading notes

Core claim

The paper's central claim is that explanation demand in human-robot interaction tracks perceived surprise, and explanation usefulness tracks the intention and key contextual factors behind the confusing action, rather than the isolated action or the abstract goal. In the user study, a one-point increase on a 5-point surprise scale multiplied the odds of wanting an explanation by 3.30 ($p < 0.001$), and the explanation type that contained only key contextual factors scored highest in usefulness (mean 4.38 on a 5-point scale, std 0.87), ahead of explanations listing beliefs, intentions, goals, actions, or context plus a redundant action. The algorithmic translation is: an action is explainable when it falls outside the expected successor set $N(a_{i-1})$ of the previous action, and the explanation content is $E(a_i) = (\Gamma_i, (a_i, \ldots, a_k))$, where $\Gamma_i$ is the union of context conditions of the current intention and all parent intentions, and $a_k$ is the first following action whose effects change that context.

Load-bearing premise

The load-bearing premise is that human surprise at a robot action can be modeled as a binary membership test on a learned set of expected action successors, because Algorithm 1 uses exactly that test to decide when to explain, and the paper never validates the mapping against human judgments of surprise.

Editorial extensions

If this is right

A BDI robot running these algorithms would speak up only when the next action violates a learned action-successor expectation, and would stay silent on routine steps, directly addressing the annoyance concern raised by earlier work.
The generated explanation names the relevant context conditions (for example, 'the dishwasher is closed') and the upcoming actions that change them, which is the content users rated highest in the survey.
Because the expected-successor function $N$ can be initialized per user and updated online, the trigger mechanism can in principle adapt to different users' prior knowledge and task habits.
The implementation in Jason/AgentSpeak shows the algorithms can be integrated into an existing BDI interpreter without changing the agent's reasoning core.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The successor-set trigger models surprise as a one-step transition violation; a model that also accounts for cost, utility, or plan rationality would plausibly explain more of the variance in explanation demand than the 21.45% pseudo-$R^2$ reported here.
The user study measured reactions to static scenarios after the fact; whether an explanation spoken just before the surprising action is as useful in live interaction is an open extension.
The same two-stage idea, flagging deviations from learned expectations and then explaining through the active goal hierarchy, could transfer to other agent architectures by substituting their plan/context representation for BDI intentions.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper gives us two things: a new user-preference dataset for robot explanations in a kitchen scenario, and two BDI algorithms that translate the survey findings into a working mechanism. The survey result is clean and worth having: surprise predicts explanation demand (OR 3.3, p<0.001), and the EC explanation type (key contextual factors) clearly beats the alternatives (mean 4.38 vs 1.53 for EA). The direction was already suggested by prior work (e.g., [2] on surprise and [15] on context), but this is a direct, scenario-based measurement with a concrete ranking of explanation content types.

The algorithms are a sensible translation of the findings. Algorithm 1 triggers when the next action is not in an expected-successor set N(a); Algorithm 2 assembles the hierarchical context plus the upcoming action that modifies it. The pseudocode is clear, and the Jason implementation is on GitHub, which is a reproducible artifact. The authors are honest that the evaluation is a prototype, not a real robot, and that user studies of the algorithms are explicitly deferred to future work.

The soft spots are real but not fatal. First, the logistic regression treats 120 scenario ratings from 33 participants as independent; a mixed-effects or cluster-robust model would be more convincing, though with OR 3.3 and p<0.001 the main conclusion probably survives. Second, the usefulness comparisons in Table II are reported as means and SDs with no significance tests; a Friedman test or mixed ANOVA would pin down which differences are reliable. Third, the stress-test concern is fair: the survey never elicits users' expected next actions, so the mapping from surprise to 'succ not in N(ai)' is an assumption, not a measured result. The algorithm also collapses graded surprise into a binary test and is not state-dependent, so it may fire in the wrong places--as the stress-test notes, opening the dishwasher is surprising in some contexts but not others. However, the paper frames N as a simple model and leaves calibration to future work, so this is a limitation rather than an internal contradiction.

Overall, this is a solid contribution for HRI and explainable-agent readers. It deserves a serious referee; I would send it to review with a request for better statistics, significance tests on the content ratings, and a more careful caveat on the N mapping. I would cite the survey results in my own work.

Referee Report

5 major / 5 minor

Summary. The paper investigates when and what a BDI home-assistant robot should explain when its behavior deviates from user expectations. The authors conducted an online scenario-based survey (N=33 participants, 120 scenario responses) testing four hypotheses: that surprise predicts explanation demand (H1), and that explanations centered on key contextual factors are most useful (H2–H4). They report a significant logistic regression of explanation demand on surprise (OR 3.30, p<0.001) and usefulness ratings in which the 'key contextual factors' explanation (EC) scored highest. Based on these findings, the paper proposes Algorithm 1, which triggers an explanation when an action is not in an expected-successor set N, and Algorithm 2, which constructs explanation content from the hierarchical intention context and relevant subsequent actions. A Jason prototype demonstrates the algorithms in a simulated kitchen-cleaning scenario.

Significance. If the central claim holds, the paper makes a useful, lightweight contribution to explainable HRI: it gives BDI agents a principled rule for when to explain and a concrete content-selection mechanism, and it grounds both in an explicit user study. The survey's finding that users prefer concise context-focused explanations over action-only or goal-only explanations echoes and extends prior work, and the open-source Jason implementation is a practical asset. The paper is also transparent about its assumptions. However, the strength of the contribution is conditional on two gaps: the statistical analysis of the survey does not account for repeated measures, and the mapping from the measured construct of surprise to the algorithm's expected-successor test is an unvalidated assumption rather than an established result. These issues affect the load-bearing claim that the algorithms implement 'surprise-driven' explanation.

major comments (5)

[Section V-.1, Algorithm 1] The trigger mechanism is not validated against the psychological construct of surprise measured in the survey. The survey (Section IV) asks participants to rate surprise after reading a described unexpected behavior, but it never elicits the user's expected next action or a graded surprise rating at the transition level. Consequently, the binary test succ ∉ N(ai) is an unverified proxy for 'surprising.' The abstract's claim that a robot can use surprise as a trigger is therefore not directly supported by the data; at present, Algorithm 1 is an assumption layered on top of the survey result, not a consequence of it. A calibration study, or at least a clearly stated limitation, is needed before the 'when to explain' component can be accepted as effective.
[Section IV, Table I] The logistic regression treats the 120 scenario responses as independent observations, but participants answered multiple scenarios (up to six, some self-selected or randomly assigned). This repeated-measures structure means that standard errors, confidence intervals, and the p-value are likely anticonservative, and the reported OR of 3.30 may not be statistically reliable. This is a load-bearing issue because H1 is the primary empirical support for the explanation-demand trigger. The analysis should use mixed-effects logistic regression or otherwise account for participant-level clustering.
[Section IV, Table II and Discussion] The claims that EC is 'most beneficial' (confirming H3) and that ECR dilutes clarity (supporting H4) are based solely on means and standard deviations, with no significance tests reported. The differences between EC (4.383), ECR (4.061), EB (3.878), and EI (3.339) may or may not be statistically meaningful, especially with repeated measures. Without a mixed-effects ANOVA or comparable pairwise comparisons, the relative ordering of explanation types is not established, and the discussion overstates the support for H2–H4.
[Section V-.1, Algorithm 1 and surrounding text] There is an internal inconsistency in the definition of the demand check. The text states: 'the robot checks whether ai ∈ N(ai+1) (line 3) and if not, triggers an explanation,' but Algorithm 1 line 3 reads 'if succ ∉ N(ai)' with succ = ai+1. These are different conditions (whether the successor is expected given the current action versus whether the current action is expected given the successor). This ambiguity must be resolved because it changes which actions are explained, and it also affects how N is updated.
[Section V-.1 and Section VII] The algorithm collapses the 1–5 surprise scale into a binary threshold and, by adding every observed transition to N after its first occurrence, makes that transition permanently expected. This cannot model situations that remain surprising on repetition or surprise that depends on graded context (e.g., an action is surprising only when it delays the user's order). The paper defers real-robot evaluation to future work, and no human-subject evaluation of Algorithms 1 and 2 is reported. A demonstration that the trigger fires in the intended situations—for example, a comparison with human surprise ratings in the same scenario—is necessary to support the claim that the system provides explanations in 'surprising situations.'

minor comments (5)

[Section VII] The word 'intension' in 'deriving key contextual factors from hierarchical intension structure' should be 'intention.'
[Section IV heading] The heading 'S TUDY OF USER PREFERENCE' contains a spacing error; it should read 'STUDY OF USER PREFERENCE.'
[Section IV, Study Design] The description of how scenarios were assigned is unclear: participants could select how many to answer (minimum three) or complete three randomly assigned scenarios. Please clarify the exact assignment procedure, as it affects the repeated-measures analysis.
[Section III, Formal Model] The notation Eff +(at) ∪ Eff −(at) in Algorithm 2 is not defined as a set operation on L(S); consider clarifying that Eff+ and Eff− are subsets of L(S) and that the intersection with Γi tests whether any effect literal lies in the context.
[Section VI] The prototype is described as 'confirming' that the agent's explanations match the design goals, but a demonstration in a simulated environment without user evaluation is better described as a feasibility check than as confirmation of effectiveness.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the survey provides external evidence, and the algorithms are implementations whose trigger set N is an input, not a fitted output.

full rationale

The paper's central empirical claims (H1-H4) come from an online user survey with logistic regression and usefulness ratings; this is external evidence, not derived from the algorithms. Algorithm 1 uses an expected-successor function N that is initialized by hand and updated with observed transitions; N is an input parameter, so the surprise trigger 'succ ∉ N(ai)' is not fitted to the survey responses. The paper does not claim to predict the survey results from the algorithms, and the Section IV findings are not used as training data to estimate N. Algorithm 2 constructs explanations from the BDI plan hierarchy (union of parent context conditions Γi plus the action sequence that changes them); this is a design choice consistent with the survey result that EC-type explanations were rated highest, but the survey finding is not an equation that the algorithm reproduces by construction. There are no load-bearing self-citations: references are to standard BDI literature (Jason/AgentSpeak) and prior HRI/XAI work, none by the present authors invoked to justify the central claim. The main weakness is external validity — N is an unvalidated model of human surprise and no human-subject evaluation of Algorithm 1 is reported — but that is a correctness/validation gap, not circularity. No equation in the paper reduces to its own inputs or renames a fitted value as a prediction.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central mechanism rests on one free parameter, the expected-successor set N, and several domain assumptions about how human expectations and contextual factors should be formalized. The algorithms themselves are simple and transparent, but the mapping from psychological constructs to formal constructs is assumed, not empirically grounded.

free parameters (1)

expected_successor_set N(a) = empty or designer-specified pairs, updated at runtime
Algorithm 1's explanation trigger depends on membership in N(a). The set is initialized ad hoc and learned by adding the observed successor after each explanation, so the notion of surprise is entirely determined by this input model rather than by measurement.

assumptions (4)

domain assumption The BDI agent model M = ⟨S,B,G,Π⟩ with AgentSpeak-style plan selection is a valid abstraction of a household cleaning robot.
Section III introduces the formal model without empirical evidence that it captures household HRI; it is a standard but unverified modeling choice.
ad hoc to paper Human expectations of robot actions are captured by a successor function N(A) based on order-linked, co-occurring, or previously observed actions.
Section V-.1 states this directly. If real expectations are richer, e.g., based on cost or social norms, the trigger mechanism will misclassify surprising actions.
ad hoc to paper The key contextual factors in an explanation are exactly the context conditions of the intention hierarchy Γ_i.
Algorithm 2 defines explanation content as the union of parent context conditions. The survey measured preferences for EC-type explanations but did not validate that this formal construction captures what users call key contextual factors.
domain assumption Surprise raises explanation demand, and this relationship transfers from the online kitchen survey to real robot interactions.
The algorithms use surprise as the sole trigger; the paper does not test other triggers such as errors, failures, or user requests within the mechanism.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Effective Explanations for Belief-Desire-Intention Robots: When and What to Explain." pith.science (2026). https://pith.science/paper/NQ6TKL7C

@misc{pith2026250702016,
  author       = {Pith},
  title        = {Pith review of: Effective Explanations for Belief-Desire-Intention Robots: When and What to Explain},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NQ6TKL7C}},
  note         = {Machine review of arXiv:2507.02016}
}

read the original abstract

When robots perform complex and context-dependent tasks in our daily lives, deviations from expectations can confuse users. Explanations of the robot's reasoning process can help users to understand the robot intentions. However, when to provide explanations and what they contain are important to avoid user annoyance. We have investigated user preferences for explanation demand and content for a robot that helps with daily cleaning tasks in a kitchen. Our results show that users want explanations in surprising situations and prefer concise explanations that clearly state the intention behind the confusing action and the contextual factors that were relevant to this decision. Based on these findings, we propose two algorithms to identify surprising actions and to construct effective explanations for Belief-Desire-Intention (BDI) robots. Our algorithms can be easily integrated in the BDI reasoning process and pave the way for better human-robot interaction with context- and user-specific explanations.

Figures

Figures reproduced from arXiv: 2507.02016 by the authors.

**Figure 2.** Illustrative questions from the questionnaire. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Distribution of usefulness scores for six explanation [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

23 extracted references · 22 canonical work pages

[2]

When do people want an explanation from a robot?

L. Wachowiak, A. Fenn, H. Kamran, A. Coles, O. Celiktutan, and G. Canal, “When do people want an explanation from a robot?” in ACM/IEEE International conference on HRI , 2024, pp. 752–761

work page 2024
[15]

Explainable ai for robot failures: Generating explanations that improve user assistance in fault recovery,

D. Das, S. Banerjee, and S. Chernova, “Explainable ai for robot failures: Generating explanations that improve user assistance in fault recovery,” in ACM/IEEE international conference on human-robot interaction, 2021, pp. 351–360

work page 2021
[1]

Explain- able agents and robots: Results from a systematic literature review,

S. Anjomshoae, A. Najjar, D. Calvaresi, and K. Fr ¨amling, “Explain- able agents and robots: Results from a systematic literature review,” in International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2019, pp. 1078–1088

work page 2019
[3]

Bdi agent programming in agentspeak using jason,

R. H. Bordini and J. F. H ¨ubner, “Bdi agent programming in agentspeak using jason,” in Computational Logic in Multi-Agent Systems , 2006, pp. 143–164

work page 2006
[4]

Explainable goal-driven agents and robots-a comprehensive review,

F. Sado, C. K. Loo, W. S. Liew, M. Kerzel, and S. Wermter, “Explainable goal-driven agents and robots-a comprehensive review,” ACM Computing Surveys , vol. 55, no. 10, pp. 1–41, 2023

work page 2023
[5]

Explainable autonomous robots: a survey and perspective,

T. Sakai and T. Nagai, “Explainable autonomous robots: a survey and perspective,” Advanced Robotics, vol. 36, no. 5-6, pp. 219–238, 2022

work page 2022
[6]

B. F. Malle, How the mind explains behavior: Folk explanations, meaning, and social interaction . MIT press, 2006

work page 2006
[7]

The category of the mind: Folk psychology of belief, desire, and intention,

Y . Kashima, A. McKintyre, and P. Clifford, “The category of the mind: Folk psychology of belief, desire, and intention,” Asian Journal of Social Psychology, vol. 1, no. 3, pp. 289–313, 1998

work page 1998

Show all 23 references

[8]

Explanation in artificial intelligence: Insights from the social sciences,

T. Miller, “Explanation in artificial intelligence: Insights from the social sciences,” Artificial intelligence, vol. 267, pp. 1–38, 2019

2019
[9]

Mental state attribution to robots: A systematic review of conceptions, methods, and findings,

S. Thellman, M. De Graaf, and T. Ziemke, “Mental state attribution to robots: A systematic review of conceptions, methods, and findings,” ACM Tran. on Hum.-Robot Interact. , vol. 11, no. 4, pp. 1–51, 2022

2022
[10]

Robot task planning and explanation in open and uncertain worlds,

M. Hanheide, M. G ¨obelbecker, G. S. Horn, A. Pronobis, K. Sj ¨o¨o, A. Aydemir, P. Jensfelt, C. Gretton, R. Dearden, M. Janicek et al. , “Robot task planning and explanation in open and uncertain worlds,” Artificial Intelligence, vol. 247, pp. 119–150, 2017

2017
[11]

Joint mind modeling for explanation generation in complex human-robot collaborative tasks,

X. Gao, R. Gong, Y . Zhao, S. Wang, T. Shu, and S.-C. Zhu, “Joint mind modeling for explanation generation in complex human-robot collaborative tasks,” in IEEE international conference on robot and human interactive communication (RO-MAN) , 2020, pp. 1119–1126

2020
[12]

Joint inference of states, robot knowledge, and human (false-) be- liefs,

T. Yuan, H. Liu, L. Fan, Z. Zheng, T. Gao, Y . Zhu, and S.-C. Zhu, “Joint inference of states, robot knowledge, and human (false-) be- liefs,” in IEEE International Conference on Robotics and Automation (ICRA), 2020, pp. 5972–5978

2020
[13]

Towards human–robot teaming: Tradeoffs of explanation-based communication strategies in a virtual search and rescue task,

E. K. Chiou, M. Demir, V . Buchanan, C. C. Corral, M. R. Endsley, G. J. Lematta, N. J. Cooke, and N. J. McNeese, “Towards human–robot teaming: Tradeoffs of explanation-based communication strategies in a virtual search and rescue task,” International Journal of Social Robotics...

2022
[14]

Towards a theory of explanations for human–robot collaboration,

M. Sridharan and B. Meadows, “Towards a theory of explanations for human–robot collaboration,” KI-K¨unstliche Intelligenz, vol. 33, no. 4, pp. 331–342, 2019

2019
[16]

Design and evaluation of explainable bdi agents,

M. Harbers, K. Van Den Bosch, and J.-J. Meyer, “Design and evaluation of explainable bdi agents,” in IEEE/WIC/ACM international conference on web intelligence and intelligent agent technology, vol. 2, 2010, pp. 125–132

2010
[17]

The “why did you do that?

V . J. Koeman, L. A. Dennis, M. Webster, M. Fisher, and K. Hindriks, “The “why did you do that?” button: Answering why-questions for end users of robotic systems,” in International Workshop on Engineering Multi-Agent Systems. Springer, 2019, pp. 152–172

2019
[18]

Explaining bdi agent behaviour through dialogue,

L. A. Dennis and N. Oren, “Explaining bdi agent behaviour through dialogue,” Auton. Agents and Multi-Agent Syst., vol. 36(2), p. 29, 2022

2022
[19]

Modeling a conversational agent using bdi framework,

A. Y . Ichida and F. Meneguzzi, “Modeling a conversational agent using bdi framework,” in ACM/SIGAPP Symposium on Applied Computing , 2023, pp. 856–863

2023
[20]

Why bad coffee? explaining bdi agent behaviour with valuings,

M. Winikoff, G. Sidorenko, V . Dignum, and F. Dignum, “Why bad coffee? explaining bdi agent behaviour with valuings,” Artificial Intelligence, vol. 300, p. 103554, 2021

2021
[21]

Evaluating a mechanism for explain- ing bdi agent behaviour,

M. Winikoff and G. Sidorenko, “Evaluating a mechanism for explain- ing bdi agent behaviour,” inInt. Workshop on Explainable, Transparent Autonomous Agents and Multi-Agent Systems , 2023, pp. 18–37

2023
[22]

Towards a multi- level explainability framework for engineering and understanding bdi agent systems

E. Yan, S. Burattini, J. F. H ¨ubner, A. Ricci et al., “Towards a multi- level explainability framework for engineering and understanding bdi agent systems.” in WOA, 2023, pp. 216–231

2023
[23]

Agentspeak(l): Bdi agents speak out in a logical com- putable language,

A. S. Rao, “Agentspeak(l): Bdi agents speak out in a logical com- putable language,” in Agents Breaking Away , W. Van de Velde and J. W. Perram, Eds., 1996, pp. 42–55

1996

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [2]

When do people want an explanation from a robot?

L. Wachowiak, A. Fenn, H. Kamran, A. Coles, O. Celiktutan, and G. Canal, “When do people want an explanation from a robot?” in ACM/IEEE International conference on HRI , 2024, pp. 752–761

work page 2024

[2] [15]

Explainable ai for robot failures: Generating explanations that improve user assistance in fault recovery,

D. Das, S. Banerjee, and S. Chernova, “Explainable ai for robot failures: Generating explanations that improve user assistance in fault recovery,” in ACM/IEEE international conference on human-robot interaction, 2021, pp. 351–360

work page 2021

[3] [1]

Explain- able agents and robots: Results from a systematic literature review,

S. Anjomshoae, A. Najjar, D. Calvaresi, and K. Fr ¨amling, “Explain- able agents and robots: Results from a systematic literature review,” in International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2019, pp. 1078–1088

work page 2019

[4] [3]

Bdi agent programming in agentspeak using jason,

R. H. Bordini and J. F. H ¨ubner, “Bdi agent programming in agentspeak using jason,” in Computational Logic in Multi-Agent Systems , 2006, pp. 143–164

work page 2006

[5] [4]

Explainable goal-driven agents and robots-a comprehensive review,

F. Sado, C. K. Loo, W. S. Liew, M. Kerzel, and S. Wermter, “Explainable goal-driven agents and robots-a comprehensive review,” ACM Computing Surveys , vol. 55, no. 10, pp. 1–41, 2023

work page 2023

[6] [5]

Explainable autonomous robots: a survey and perspective,

T. Sakai and T. Nagai, “Explainable autonomous robots: a survey and perspective,” Advanced Robotics, vol. 36, no. 5-6, pp. 219–238, 2022

work page 2022

[7] [6]

B. F. Malle, How the mind explains behavior: Folk explanations, meaning, and social interaction . MIT press, 2006

work page 2006

[8] [7]

The category of the mind: Folk psychology of belief, desire, and intention,

Y . Kashima, A. McKintyre, and P. Clifford, “The category of the mind: Folk psychology of belief, desire, and intention,” Asian Journal of Social Psychology, vol. 1, no. 3, pp. 289–313, 1998

work page 1998