REVIEW 4 major objections 3 minor 43 references
Counterfactual Explanations as Plans
T0 review · 4 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Counterfactual explanations of plans are minimally different alternative plans that flip the outcome, and model reconciliation is the same idea in epistemic form.
desk verdict A clean conceptual framework for counterfactual explanations in planning, but the central unification claim rests on under-specified minimality conditions that need to be made precise before the definitions are implementable. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the counterfactual plan $\delta'$ of Definition 1, chosen to be minimally distant from the original $\delta$ while making the goal false; distance is made concrete by length-based minimality (Definition 2), fluent-based minimality (Definition 4), or the joint plan-and-effect measure (Definition 6). In the reconciliation settings the same counterfactual template is applied with epistemic goals: the missing ingredient is a formula $\alpha$ added to the agent's initial theory (or a false belief $\beta$ removed) such that the agent comes to know $\varphi$, with 'smallest such $\alpha$' playing the role of minimal distance. The supporting machinery is the modal logic ES—a situation-calculus-style language with action modalities $[a]$, 'always' $2$, knowledge $K$, and only-knowing $O$—together with basic action theories, successor state and sensing axioms, and the distinction between the real world $\Sigma_0$ and the agent's believed theory $\Sigma'_0$.
What would settle it
Build a basic action theory where achievable plan distances have no minimum (for example, an infinite descending chain generated by inserting ever-shorter no-op sequences) or where two incomparable formulas both restore the agent's knowledge of the goal; either case would show the definitions do not always pick out a unique explanation.
Extended reading notes
Core claim
The paper's central claim is that counterfactual explanations generalize from single decisions to action sequences: given a plan $\delta$ that entails goal $\varphi$, a counterfactual explanation is a plan $\delta'$ with $\Sigma \models \mathrm{Exec}(\delta') \land [\delta']\neg\varphi$ and $\mathrm{dist}(\delta',\delta)$ minimal (Definition 1). The paper then shows that model reconciliation is the same counterfactual idea once one separates truth from belief: the user either supplies a missing formula $\alpha$ or replaces a false belief $\beta$ so that, after a minimally distant plan, the agent knows the goal (Definitions 13, 16, 18, 21). This is formalized in the modal epistemic logic ES, where the agent's uncertainty is modeled by a set of possible worlds and the only-knowing operator $O$ captures both beliefs and non-beliefs.
Load-bearing premise
The framework assumes that for every explanation query there is a well-defined 'closest' alternative plan or 'smallest' piece of missing knowledge, but the definitions do not fix a distance metric or an ordering on formulas, so in some cases no unique explanation exists.
Editorial extensions
If this is right
- Contrastive explanations in planning—where the answer to 'why this action rather than that one' is an altered action sequence—become a special case of Definition 1.
- Model reconciliation, where the user corrects the agent's model or suggests actions, is captured by the same counterfactual template, with the smallest missing formula or false belief as the explanation.
- Because sensing actions are allowed, a counterfactual explanation can be a plan that gains information rather than only changing the world.
- The paper's claim implies that existing discrepancy and skeptical/credulous reconciliation accounts can be re-expressed as variations of Definitions 13 through 21.
- Since reasoning in ES can reduce to non-modal reasoning, the definitions are in principle implementable with standard planning and regression techniques.
Reading between the lines
- The choice of distance metric is decisive: length, affected fluents, and action costs can rank the same pair of plans differently, so the framework predicts that what counts as a good explanation depends on the user's implicit notion of closeness; this is testable by user studies.
- The paper treats the user's knowledge as a proxy for ground truth in a single-agent setting, which suggests a natural extension to genuine multi-agent counterfactual explanations by adding separate epistemic operators for each agent.
- In machine learning, a 'plan' could be an ordered sequence of feature changes, which would connect this framework to algorithmic recourse; the paper does not explore that application.
- One could derive concrete algorithms by instantiating the distance metric with plan cost and using a planner to search for the minimally distant goal-flipping sequence, then compare the generated explanations with human judgments.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper develops a formal account of counterfactual explanations in the modal situation calculus ES. It defines a counterfactual explanation of a goal φ after an executable plan δ as an alternative executable plan δ′ that makes ¬φ true and is minimally distant from δ (Definition 1), and then extends this to epistemic goals (Definition 11), missing actions, missing knowledge, weakened truths, false beliefs (Definitions 13, 16, 18, 21), and diverse explanations (Definitions 8 and 10). The paper claims that contrastive explanations and model reconciliation become instances of this single counterfactual-plan framework. The formal content is definitional and example-driven; there are no theorems or proofs.
Significance. If the framework were made fully precise and the claimed links to model reconciliation and contrastive explanations were proved, this would be a useful unification: it gives an epistemic-logic account of counterfactual plans and clarifies how truth and belief interact in explanation. The use of ES and only-knowing is a good fit for distinguishing the user's model from the agent's beliefs, and the diversity and possibility variants show the approach has breadth. At present, however, the load-bearing notions of minimal distance and smallest knowledge addition are left unspecified, and the unification claims are asserted rather than proved. The paper is a promising formalization program rather than a completed formal result.
major comments (4)
- [Section 4, Definition 1 (and Definitions 11, 13, 18)] The defining condition 'dist(δ′,δ) is minimal' requires a fixed distance function and a guarantee that the minimization is attained, but neither is supplied. The metrics sketched in Definitions 2, 4, and 6 are presented only as examples, so every subsequent definition remains parameterized by an unspecified dist. For arbitrary Σ, δ, φ, the set of eligible δ′ may be empty, and with a real-valued cost the infimum may not be attained. Please state the definitions relative to a chosen metric with an explicit well-foundedness or existence condition, or prove that a minimizer exists under the stated assumptions.
- [Section 5.1, Definition 16 (and Definitions 18, 21)] The phrase 'the smallest such α' is not defined. Under the natural ordering by logical strength, a smallest α need not exist: if α is sufficient, any α′ logically stronger than α is also sufficient, so the set of sufficient formulas is upward closed and may have no least element in a first-order setting. Moreover, restricting α to a single sentence in Σ0−Σ′0 makes the definition empty when the missing knowledge is a conjunction of two facts. The paper should specify an ordering on formulas, allow finite conjunctions or sets, and prove existence (or state conditions under which a minimum exists).
- [Sections 1 and 5.4] The central claim that existing accounts of discrepancy, contrastive explanations, and credulous/skeptical reconciliation are variations of this framework is not made precise. No theorem or translation shows that, for example, the reconciliation definitions of Section 5 correspond to the discrepancy notion of [37] or that the B-based variant of Section 5.4 corresponds to credulous entailment in [42]. Without an explicit correspondence result, the claimed unification is an informal analogy rather than a formal contribution.
- [Section 5.1, Definition 13] The step from 'Σ ̸|= [δ]Kφ' to 'Σ |= [δ]¬Kφ' is not justified by the semantics in Section 2. In general, failure of entailment means there is a model where Kφ is false, not that Kφ is false in every model. The only-knowing operator can yield negative knowledge under suitable conditions, but the paper does not state or prove the needed property for formulas of the form [δ]Kφ. This makes the trigger condition of Definition 13, and the analogous reasoning in Examples 14 and 15, unsupported as written.
minor comments (3)
- [Section 4, Example 9] Example 9 is inconsistent with Definition 8. If φ = ∃x¬Broken(x), then ¬φ = ∀xBroken(x), so α ∧ ¬φ is ∃xGlass(x) ∧ ∀xBroken(x), not ∃x(Glass(x) ∧ Broken(x)). The proposed δ′ = pickup(c) · drop(c) does not make all objects broken in the example domain. Please modify either φ, α, or the example so that the formal condition and the worked scenario agree.
- [Section 4, Definition 10] Definition 10 is titled 'diverse CF explanations' but contains no diversity condition: it merely collects sequences satisfying the distance bound. As written, the set could contain one element or many identical elements, so the term 'diverse' is not justified by the formal condition. A diversity measure or an explicit constraint on the set should be added.
- [Section 3] There is a typo in the sentence 'We will included an extended report with some examples'; it should read 'We will include an extended report'.
Circularity Check
No significant circularity: the counterfactual/reconciliation definitions are stipulated formalizations, not fitted predictions, and the few self-citations are not load-bearing.
full rationale
The paper's central move is to define counterfactual explanations as minimally distant alternative action sequences that toggle a goal (Definition 1) and then to reuse the same schema, with epistemic goals and model updates, in Definitions 13, 16, 18 and 21. This is an act of definitional unification rather than a derivation of an empirical result from fitted inputs: the reconciliation accounts are stipulated to be instances of the counterfactual schema, and the paper explicitly claims the relationship 'can be seen as a counterfactual' rather than deriving it from independent premises. No parameter is fitted to data and then renamed as a prediction; no quantity is defined in terms of the quantity it is claimed to explain. The logic ES is imported from prior work by Lakemeyer and Levesque ([23]), not from the author's own papers, and the paper explicitly says 'Our ideas do not hinge on this language, and so any planning language that helps us reason about truth, knowledge, actions and sensing should suffice.' The only self-citations ([3], [30]) are pointer-style references to multi-agent extensions and epistemic-planning implementations; they do not carry the load of the formal definitions. The reviewer-identified weakness—that 'dist minimal' and 'smallest such alpha' are left with unspecified metrics or orderings and no existence proof—is a well-definedness and completeness gap, not circularity: an ill-posed optimization does not make the definition equivalent to its inputs. Likewise, the omitted proof of regression and representation theorems (Section 3, 'We omit the details but refer readers to [23]') is an external citation to independent prior work, and the paper's own admission that an extended report with examples is pending is a presentation gap. Overall, the derivation chain is self-contained at the level of definitions; the main risk is formal adequacy, not circularity.
Assumptions & free parameters
free parameters (3)
- distance metric (dist) =
unspecified; candidates: length, fluent-set, plan-and-effect
- diversity bound k =
positive integer, user-specified
- "smallest α" ordering =
undefined
assumptions (6)
- domain assumption User knowledge can stand in for ground truth in the real world
- domain assumption Agent believes the dynamics exactly as the real world does
- domain assumption Initial agent knowledge is a subset or weakening of world facts (except false belief section)
- ad hoc to paper Existence of minimal distances and smallest knowledge additions
- standard math Standard first-order logic with equality, unique names, standard names and infinitary domain closure
- standard math Modal logic ES properties, including Barcan formula and only-knowing characterization
Cite this review
Pith. "Pith review of Counterfactual Explanations as Plans." pith.science (2026). https://pith.science/paper/I5RNUQHO
@misc{pith2026250209205,
author = {Pith},
title = {Pith review of: Counterfactual Explanations as Plans},
year = {2026},
howpublished = {\url{https://pith.science/paper/I5RNUQHO}},
note = {Machine review of arXiv:2502.09205}
}
read the original abstract
There has been considerable recent interest in explainability in AI, especially with black-box machine learning models. As correctly observed by the planning community, when the application at hand is not a single-shot decision or prediction, but a sequence of actions that depend on observations, a richer notion of explanations are desirable. In this paper, we look to provide a formal account of ``counterfactual explanations," based in terms of action sequences. We then show that this naturally leads to an account of model reconciliation, which might take the form of the user correcting the agent's model, or suggesting actions to the agent's plan. For this, we will need to articulate what is true versus what is known, and we appeal to a modal fragment of the situation calculus to formalise these intuitions. We consider various settings: the agent knowing partial truths, weakened truths and having false beliefs, and show that our definitions easily generalize to these different settings.
Reference graph
Works this paper leans on
-
[37]
In: Proceedings of the International Conference on Automated Planning and Scheduling, 32, pp
Maayan Shvo, Toryn Q Klassen & Sheila A McIlraith (2022): Resolving Misconceptions about the Plans of Agents via Theory of Mind. In: Proceedings of the International Conference on Automated Planning and Scheduling, 32, pp. 719–729
work page 2022
-
[42]
Journal of Artificial Intelligence Research 73, pp
Stylianos Loukas Vasileiou, William Yeoh, Tran Cao Son, Ashwin Kumar, Michael Cashmore & Dianele Magazzeni (2022): A Logic-Based Explanation Generation Framework for Classical and Hybrid Planning Problems. Journal of Artificial Intelligence Research 73, pp. 1473–1534, doi:10.1613/jair.1.13431
-
[1]
C. Baral & M. Gelfond (2005): Logic programming and reasoning about actions. In: Handbook of Temporal Reasoning in Artificial Intelligence, Elsevier, pp. 389–426, doi:10.1016/S1574-6526(05)80015-X
-
[2]
Chitta Baral, Thomas Bolander, Hans van Ditmarsch & Sheila McIlrath (2017): Epistemic Planning (Dagstuhl Seminar 17231) . Dagstuhl Reports 7(6), pp. 1–47, doi:10.4230/DagRep.7.6.1. Available at http://drops.dagstuhl.de/opus/volltexte/2017/8285
-
[3]
V . Belle & G. Lakemeyer (2014): Multiagent Only Knowing in Dynamic Systems . Journal of Artificial Intelligence Research 49, doi:10.1613/jair.4192
-
[4]
Theory and Practice of Logic Programming, pp
Leopoldo Bertossi (2021): Declarative approaches to counterfactual explanations for classification. Theory and Practice of Logic Programming, pp. 1–35, doi:10.1017/s1471068421000582
-
[5]
Theory and Practice of Logic Programming 20(6), pp
Aysu Bogatarkan & Esra Erdem (2020): Explanation generation for multi-modal multi-agent path finding with optimal resource utilization using answer set programming. Theory and Practice of Logic Programming 20(6), pp. 974–989, doi:10.1017/S1471068420000320
-
[6]
arXiv preprint arXiv:1810.06338
Rita Borgo, Michael Cashmore & Daniele Magazzeni (2018): Towards providing explanations for AI planner decisions. arXiv preprint arXiv:1810.06338
arXiv 2018
Show all 43 references
-
[7]
arXiv preprint arXiv:1908.05059
Michael Cashmore, Anna Collins, Benjamin Krarup, Senka Krivic, Daniele Magazzeni & David Smith (2019): Towards explainable AI planning as a service. arXiv preprint arXiv:1908.05059
2019 arXiv
-
[8]
In: IJCAI, pp
Jens Classen, Patrick Eyerich, Gerhard Lakemeyer & Bernhard Nebel (2007): Towards an Integration of Golog and Planning. In: IJCAI, pp. 1846–1851. 166 Counterfactual Explanations as Plans
2007
-
[9]
In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp
Xinyue Dai, Mark T Keane, Laurence Shalloo, Elodie Ruelle & Ruth MJ Byrne (2022): Counterfactual explanations for prediction and diagnosis in xai . In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 215–226, doi:10.1145/3514094.3534144
2022
-
[10]
Delgrande & Hector J
James P. Delgrande & Hector J. Levesque (2012): Belief Revision with Sensing and Fallible Actions . In: Proc. KR
2012
-
[11]
H. P. van Ditmarsch, A. Herzig & T. De Lima (2011): From Situation Calculus to Dynamic Epistemic Logic. J. Log. Comput. 21(2), pp. 179–204, doi:10.1093/logcom/exq024
2011 doi
-
[12]
The Review of Symbolic Logic 8(1), pp
Jie Fan, Yanjing Wang & Hans Van Ditmarsch (2015): Contingency and knowing whether. The Review of Symbolic Logic 8(1), pp. 75–107, doi:10.1017/S1755020314000343
2015 doi
-
[13]
Theory and Practice of Logic Programming 22(6), pp
Jorge Fandinno, Wolfgang Faber & Michael Gelfond (2022): Thirty years of epistemic specifications. Theory and Practice of Logic Programming 22(6), pp. 1043–1083, doi:10.1017/S147106842100048X
2022 doi
-
[14]
arXiv preprint arXiv:1709.10256
Maria Fox, Derek Long & Daniele Magazzeni (2017): Explainable planning . arXiv preprint arXiv:1709.10256
2017 arXiv
-
[15]
Baier & Sheila A
Christian Fritz, Jorge A. Baier & Sheila A. McIlraith (2008): ConGolog, Sin Trans: Compiling ConGolog into Basic Action Theories for Planning and Beyond . In Brewka & Lang, editors: Principles of Knowledge Representation and Reasoning: Proceedings of the Eleventh International...
2008
-
[16]
Gelfond & V
M. Gelfond & V . Lifschitz (1993): Representing action and change by logic programs. The Journal of Logic Programming 17(2-4), pp. 301–321, doi:10.1016/0743-1066(93)90035-F
1993 doi
-
[17]
Artificial intelligence 30(1), pp
Matthew L Ginsberg (1986): Counterfactuals. Artificial intelligence 30(1), pp. 35–79, doi:10.1016/0004- 3702(86)90067-6
1986 doi
-
[18]
Halpern & Yoram Moses (1990): Knowledge and Common Knowledge in a Distributed Environ- ment
Joseph Y . Halpern & Yoram Moses (1990): Knowledge and Common Knowledge in a Distributed Environ- ment. J. ACM 37(3), pp. 549–587, doi:10.1145/79147.79161
1990
-
[19]
AI Magazine 41(3), doi:10.1609/aimag.v41i3.5257
Subbarao Kambhampati (2020): Challenges of Human-Aware AI Systems. AI Magazine 41(3), doi:10.1609/aimag.v41i3.5257
2020 doi
-
[20]
R. F. Kelly & A. R. Pearce (2008): Complex Epistemic Modalities in the Situation Calculus
2008
-
[21]
ICAPS 2019 Workshop on Explainable AI Planning (XAIP)
Benjamin Krarup, Michael Cashmore, Daniele Magazzeni & Tim Miller (2019): Model-based contrastive explanations for explainable planning. ICAPS 2019 Workshop on Explainable AI Planning (XAIP)
2019
-
[22]
Lakemeyer & H
G. Lakemeyer & H. J. Levesque (2011): A semantic characterization of a useful fragment of the situation calculus with knowledge. Artificial Intelligence 175, pp. 142–164, doi:10.1016/j.artint.2010.04.005
2011 doi
-
[23]
Lakemeyer & Hector J
G. Lakemeyer & Hector J. Levesque (2004): Situations, Si! Situation terms, No! In: Proc. KR, pp. 516–526
2004
-
[24]
Lespérance, H
Y . Lespérance, H. J. Levesque, F. Lin & R. B. Scherl (2000): Ability and Knowing How in the Situation Calculus. Studia Logica 66(1), pp. 165–186, doi:10.1023/A:1026761331498
2000 doi
-
[25]
Levesque, R
H. Levesque, R. Reiter, Y . Lespérance, F. Lin & R. Scherl (1997):GOLOG: A Logic Programming Language for Dynamic Domains. Journal of Logic Programming 31, pp. 59–84, doi:10.1016/S0743-1066(96)00121-5
1997 doi
-
[26]
H. J. Levesque (1990): All I know: a study in autoepistemic logic . Artificial Intelligence 42(2-3), pp. 263– 309, doi:10.1016/0004-3702(90)90056-6
1990 doi
-
[27]
H. J. Levesque (1996): What Is Planning in the Presence of Sensing?In: Proc. AAAI / IAAI, pp. 1139–1146
1996
-
[28]
F. Lin & R. Reiter (1994): Forget it. In: Working Notes of AAAI Fall Symposium on Relevance, pp. 154–159
1994
-
[29]
In: Proceedings of the 2020 conference on fairness, account- ability, and transparency, pp
Ramaravind K Mothilal, Amit Sharma & Chenhao Tan (2020): Explaining machine learning classifiers through diverse counterfactual explanations. In: Proceedings of the 2020 conference on fairness, account- ability, and transparency, pp. 607–617, doi:10.1145/3351095.3372850
2020
-
[30]
Muise, V
C. Muise, V . Belle, P. Felli, S. McIlraith, T. Miller, A. Pearce & L. Sonenberg (2015):Planning Over Multi- Agent Epistemic States: A Classical Planning Approach. In: Proc. AAAI, doi:10.1609/aaai.v29i1.9665
2015 doi
-
[31]
Cambridge university press, doi:10.1017/CBO9780511803161
Judea Pearl (2009): Causality. Cambridge university press, doi:10.1017/CBO9780511803161. Vaishak Belle 167
2009 doi
-
[32]
Pednault (1989): ADL: Exploring the middle ground between STRIPS and the situation calculus
E.P.D. Pednault (1989): ADL: Exploring the middle ground between STRIPS and the situation calculus . In: Proc. KR, pp. 324–332, doi:10.5555/112922.112954
1989
-
[33]
In: Joint German/Austrian Conference on Artificial Intelligence (Künstliche Intelligenz), Springer, pp
Daniel Reifsteck, Thorsten Engesser, Robert Mattmüller & Bernhard Nebel (2019): Epistemic multi-agent planning using monte-carlo tree search . In: Joint German/Austrian Conference on Artificial Intelligence (Künstliche Intelligenz), Springer, pp. 277–289, doi:10.1007/978-3-030...
2019 doi
-
[34]
Reiter (2001): Knowledge in action: logical foundations for specifying and implementing dynamical systems
R. Reiter (2001): Knowledge in action: logical foundations for specifying and implementing dynamical systems. MIT Press, doi:10.7551/mitpress/4074.001.0001
2001 doi
-
[35]
R. B. Scherl & H. J. Levesque (2003): Knowledge, action, and the frame problem . Artificial Intelligence 144(1-2), pp. 1–39, doi:10.1016/S0004-3702(02)00365-X
2003 doi
-
[36]
Shapiro, Y
S. Shapiro, Y . Lespérance & H.J. Levesque (2002): The cognitive agents specification language and verifi- cation environment for multiagent systems. In: Proc. AAMAS, pp. 19–26, doi:10.1145/544741.544746
2002
-
[38]
In: Twelfth International Conference on the Principles of Knowledge Representation and Reasoning
Shirin Sohrabi, Jorge A Baier & Sheila A McIlraith (2010): Diagnosis as planning revisited . In: Twelfth International Conference on the Principles of Knowledge Representation and Reasoning
2010
-
[39]
T.C. Son & C. Baral (2001): Formalizing sensing actions–A transition function based approach . Artificial Intelligence 125(1-2), pp. 19–91, doi:10.1016/S0004-3702(00)00080-1
2001 doi
-
[40]
In: Proceedings of the International Conference on Automated Planning and Scheduling, 28, pp
Sarath Sreedharan, Tathagata Chakraborti & Subbarao Kambhampati (2018): Handling model uncertainty and multiplicity in explanations via model reconciliation. In: Proceedings of the International Conference on Automated Planning and Scheduling, 28, pp. 518–526
2018
-
[41]
Siddharth Srivastava (2010): Foundations and Applications of Generalized Planning. Ph.D. thesis, Depart- ment of Computer Science, University of Massachusetts Amherst, doi:10.3233/AIC-2011-0508
2010 doi
-
[43]
Sandra Wachter, Brent Mittelstadt & Chris Russell (2017): Counterfactual explanations without opening the black box: Automated decisions and the GDPR. Harv. JL & Tech.31, p. 841
2017
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.