Pith. sign in

REVIEW 3 major objections 4 minor 82 references

Argumentative Ensembling for Robust Recourse under Model Multiplicity

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that under model multiplicity, choosing which models to ensemble and which counterfactual explanations to return must be done together, and proves an argumentation-based method that guarantees every returned…

desk verdict Solid incremental extension of the authors' AAMAS 2024 argumentative ensembling framework, with real theoretical content, but the abstract overstates the robustness guarantee and the experiments leak test labels into the preference construction. read the letter →

arxiv 2506.20260 v1 pith:FC3ZZJXM submitted 2025-06-25 cs.LG cs.AIcs.MA

Junqi Jiang , Antonio Rago , Francesco Leofante , Francesca Toni This is my paper
classification cs.LGcs.AIcs.MA
keywords modelmultiplicitycounterfactualexplanationsalgorithmicrecoursecomputationalargumentationbipolarframeworksensemblingargumentativerecourse-aware
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

When several equally accurate models disagree on the same input, a counterfactual explanation computed for one model may be invalid for another, so recourse advice becomes unreliable. This paper formalises the problem of choosing which models to aggregate and which counterfactuals to return at the same time, which it calls recourse-aware ensembling (RAE), and proposes six properties that a good solution should satisfy. It shows that the natural extensions of majority-vote ensembling fail these properties, and it introduces argumentative ensembling, which builds a bipolar argumentation framework in which models attack models that disagree with them and counterfactuals attack models on which they fail. The paper's central claim is that with the s-preferred argumentation semantics, this method guarantees that at least one model and one counterfactual are returned, all returned models agree, every returned counterfactual is valid on every returned model, and a model is returned exactly when its own counterfactual is returned. If true, this gives a principled way to provide robust recourse recommendations under model multiplicity while still letting users express preferences over which models should win.

What carries the argument

The central object is the Bipolar Argumentation Framework (BAF), a directed graph whose nodes are the models and the counterfactuals, whose attack edges represent conflicts (two models disagreeing on the input prediction, or a counterfactual failing to change a model's prediction), and whose support edges pair each model with its own counterfactual. Model preferences are folded into the attack relation: a less-preferred model can only be attacked, while a more-preferred attacker is always present. Argumentation semantics, specifically the safe-preferred (s-preferred) semantics, which requires the chosen set to be conflict-free, safe, and to defend all its members, select the cardinality-maximal extension, and that extension is the ensemble solution. The support edges are what carry the coherence guarantee: because a model and its counterfactual are supported only by each other, an indirect attack on one is equivalent to a direct attack on the other, which forces the s-preferred extension to contain them together or not at all.

What would settle it

Run s-preferred argumentative ensembling on an instance where one model's own counterfactual is invalid on that model, i.e., $M_i(c_i) = M_i(x)$, and check whether the returned set still contains $M_i$ exactly when it contains $c_i$; the paper's proof of counterfactual coherence depends on the mutual support between $M_i$ and $c_i$, so this case is where the guarantee would fail.

Watch

Extended reading notes

Core claim

The paper's central claim is Theorem 5: s-preferred argumentative ensembling satisfies non-emptiness, model agreement, counterfactual validity, and counterfactual coherence, and it satisfies non-triviality under a stated condition. In concrete terms, the ensemble always contains at least one model and one counterfactual, every selected model gives the same prediction for the input, every returned counterfactual flips the prediction of every returned model ($M_i(c_j) \neq M_i(x)$ for all selected $M_i$ and $c_j$), and a model appears in the solution if and only if the counterfactual computed for that model also appears. The method achieves this by representing each model and each counterfactual as an argument, letting a model attack another model when their predictions differ, letting a counterfactual and a model attack each other when the counterfactual does not flip that model's prediction, and letting each model and its own counterfactual support each other; the s-preferred semantics then selects a maximal set of arguments that is safe and defends itself. The paper also shows that stable semantics coincide with d-preferred semantics and c-preferred with s-preferred under this construction, and it demonstrates empirically on three datasets that the guarantees hold in practice while preferences over model accuracy or simplicity steer the ensemble.

Load-bearing premise

The load-bearing premise is that every model already has a counterfactual explanation that is valid on that model itself; if any counterfactual generator returns an invalid counterfactual for its own model, the support structure and with it the counterfactual-coherence guarantee break down.

Editorial extensions

If this is right

  • Under the s-preferred semantics, recourse-aware ensembling always returns a non-empty set of models and a non-empty set of counterfactuals, so the user is never left without a recommendation.
  • Every counterfactual returned by the s-preferred instantiation is valid for every model in the returned ensemble, which is exactly the robustness under model multiplicity that the paper advertises.
  • Counterfactual coherence ties each returned counterfactual to its source model: asking for a model's explanation brings the model along, and choosing the model brings its explanation.
  • User preferences over model properties such as accuracy and simplicity can be injected through the attack relation, and the empirical results show that preference-aware instantiations steer the ensemble toward the preferred property without sacrificing prediction accuracy.
  • The price of these guarantees is majority vote: argumentative ensembling can return a minority prediction when the majority's counterfactuals are invalid, although empirically the accuracy remains competitive.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper assumes every model already has a valid counterfactual; a natural extension the authors leave open is to treat invalid counterfactuals as attackers rather than supporters, which would let the same argumentation machinery repair broken counterfactuals instead of requiring them upfront.
  • Because the paper proves that stable equals d-preferred and c-preferred equals s-preferred for this construction, the four semantics collapse into two distinct behaviours, so a practitioner only needs to choose between the permissive d-preferred and the strict s-preferred family.
  • The tie-breaking rule used in the implementation, choosing among cardinality-maximal extensions the one matching majority vote, is an editorial choice not part of the formal definition, so the method's practical behaviour can be better than its theoretical worst case.
  • The framework's guarantees are binary property checks; a continuous measure of how many models validate a counterfactual, or how close the ensemble is to the majority, would let users trade robustness against majority vote more finely.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper formalizes recourse-aware ensembling (RAE) under model multiplicity, introducing six desirable properties for solutions and showing that naive extensions of majority-vote ensembling fail some of them. It then proposes argumentative ensembling, which encodes models and counterfactual explanations as arguments in a bipolar argumentation framework and applies argumentation semantics to select an ensemble. The central theoretical claim is Theorem 5: s-preferred argumentative ensembling satisfies non-emptiness, model agreement, counterfactual validity, and counterfactual coherence, with non-triviality under a stated condition. The paper also claims equivalences between stable and d-preferred semantics and between c-preferred and s-preferred semantics, and reports experiments on three datasets with multiple preference configurations.

Significance. The problem is timely, and the paper's formal decomposition into six properties is a useful contribution regardless of the specific method. The BAF-based approach is original, and the empirical study is broad, with public code and several preference instantiations. However, the main theoretical guarantee, Theorem 5, is false: I provide a small counterexample inside the paper's own definitions in which an s-preferred extension violates counterfactual coherence. Since the paper's contribution rests on this theorem, the theoretical claims as stated cannot be accepted; the empirical results do not compensate for a false central result.

major comments (3)
  1. [Section 5.3, Theorem 5] The counterfactual-coherence claim is false, and the proof's step from ⊆-maximality to coherence is invalid. Counterexample with only valid CEs: let M={M1,M2} with M1(x)=M2(x)=0, C={c1,c2}, M1(c1)=1, M2(c2)=1, M2(c1)=0 (so c1 is invalid only on M2), and c2 valid on both models; preferences equal. Under Definition 12, supports are (M1,c1),(c1,M1),(M2,c2),(c2,M2), and the only attacks are (M2,c1),(c1,M2). The set X={M1,c2} is s-admissible: it is conflict-free and safe (M1 supported-attacks M2 via M1→c1→M2, c2 supported-attacks c1 via c2→M2→c1, and no element of X attacks or supports another element of X); it defends M1 against M2's indirect attack M2→c1→M1 by M1→c1→M2, and defends c2 against c1's indirect attack c1→M2→c2 by c2→M2→c1. No superset is s-admissible: adding M2 makes M1 attack M2 while c2 supports M2, and adding c1 makes c2 attack c1 while M1 supports c1. Thus X is an s-preferred extension, yet M1∈X while c1∉X and c2∈X while M2∉X, violating Definition 7. This is not an artifact of invalid CEs: every CE is valid on its own model.
  2. [Section 5.3, Theorem 6] The claimed equivalence between c-preferred and s-preferred argumentative ensembling is false. In the counterexample above, X={M1,c2} is an s-preferred extension but is not closed under the support relation S: M1∈X while c1∉X, and c2∈X while M2∉X. Hence X is not c-admissible. The proof's inference from Theorem 4 and 'Property 2 of [19]' does not establish that every s-admissible set is closed under S; indeed the counterexample shows that s-admissible sets need not be closed. Consequently the paper's claim that c-preferred argumentative ensembling inherits the properties of s-preferred argumentative ensembling is unsupported and, as stated, incorrect.
  3. [Abstract and Theorem 5] The headline claim 'guarantees the robustness of CEs under MM' is stated without the precondition that every input CE is valid on its own model. The formal definition of a CE in Section 3 assumes M_i(c_i)≠M_i(x), and Section 7 explicitly concedes that the method assumes a valid CE for each model. If a CE generator returns an invalid CE for its own model, Definition 12 still inserts the i=j support and mutual attack, and the s-preferred extension can be empty, so non-emptiness fails as well. The abstract and theorem statements should carry this precondition explicitly or the method must be extended to handle invalid CEs; as written, the advertised guarantee is stronger than what is proved.
minor comments (4)
  1. [Theorem 7 proof] The proof contains a typographical error: 'Mk(cl)≠Mk(cl)' should presumably read 'Mk(cl)≠Mk(x)'.
  2. [Section 6.1.4] The implementation deviates from Definition 13: when multiple cardinality-maximal extensions exist, the implementation returns one matching majority vote and, for d-preferred semantics, prefers extensions containing both models and CEs. The reported property-satisfaction rates therefore describe the implemented procedure, not the formal Definition 13 as stated, and this should be acknowledged explicitly.
  3. [Theorem 8] Theorem 8 assumes ∀ci∈C, ∀Mi∈M, Mi(ci)≠Mi(x), i.e., every CE is valid on every model, which is substantially stronger than the paper's running assumption that each ci is valid on its own Mi. Section 5.5 motivates the BAF by the possibility of invalid own-model CEs, which is inconsistent with the stated hypothesis of Theorem 8.
  4. [Section 3 and general proofreading] There are several small proofreading issues: in the definition of A(α1), the set-builder uses '∈X' where it should use '∈A'; in Theorem 5's proof there is a stray 'Definition . 11'; and Table 4 shows identical rows for Sa,d-A and Sa,s-A across all reported entries, which is not explained in the text.

Circularity Check

1 steps flagged · score 2.0 of 10

Theorem 5's counterfactual-coherence guarantee is hard-wired into Definition 12's support relation; the method is otherwise self-contained, with the invalid-CE precondition explicitly disclosed.

  1. self definitional [Definition 12 (Section 5.1) and Theorem 5 proof (Section 5.3)]
    "S ⊆ (M×C)∪(C×M) where for any Mi ∈ M and cj ∈ C, (Mi, cj), (cj, Mi)∈ S iff i= j. ... since we know any P s i ∈ P s is ⊆-maximal by Definition 13, P s i must be such that ∀i∈ {1, . . . , m}, Mi ∈ S iff ci ∈ S."

    Counterfactual coherence (Definition 7) requires exactly that Mi∈S iff ci∈S for every model–CE pair. Definition 12 builds the BAF's support relation as exactly those pairs and no other supports. Thus Theorem 5's coherence guarantee is not an emergent property of argumentation semantics; it is an immediate unpacking of the input construction. The proof itself states that these are the only support relations and then concludes the iff. This is a design choice rather than a hidden assumption, and the paper is transparent about the coupling, so the circularity is minor and localized.

full rationale

The paper is a design/theory contribution: it defines desirable properties and constructs a BAF-based ensembling method intended to satisfy them. The counterfactual-coherence guarantee in Theorem 5 is real but is baked into the representation: Definition 12 couples each model with its own CE via support, so the proof's conclusion is essentially a restatement of the construction. This is a self-definitional element, but not a deceptive one, because the authors openly motivate the design with this coupling and the s-preferred semantics still performs nontrivial selection among maximal safe sets. No fitted parameter is renamed as a prediction, and the formal results are proven in the text rather than delegated to self-citations. The paper also explicitly discloses the key precondition that every input CE is valid on its own model (Section 7), so the abstract's unqualified robustness claim is stronger than what is guaranteed for arbitrary heuristic CE generators, but the omission is a scope limitation, not circularity. Overall circularity is minor and localized to one by-construction property, giving a score of 2.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The method itself is parameter-free; the proofs rely on standard BAF semantics from [19] and Dung's argumentation [21], plus domain assumptions about valid CEs and total-order preferences. Experimental preference scores (accuracy, simplicity) are hand-chosen for the evaluation and are not load-bearing for the theoretical guarantees.

assumptions (4)
  • domain assumption Each model Mi in M is paired with a CE ci that is valid for Mi (Mi(ci) != Mi(x))
    Assumed in Section 3 and used in Definition 12 to build support relations S; the paper flags this as a limitation in Section 7.
  • domain assumption Model preferences ⪯M form a total order over M
    Definition 11 assumes a total order; used in attack definitions (Definition 12) and in proofs of non-emptiness (Theorem 5).
  • standard math BAF semantics (d-, s-, c-preferred, stable) and their properties are as defined in [19]
    Used throughout Section 5; the paper relies on [19] for equivalence properties such as 'Property 2' used in Theorem 6.
  • standard math Dung's abstract argumentation theory [21] (conflict-free, defense, preferred extensions) is standard
    Framework for AAF equivalences in Section 5.5.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Argumentative Ensembling for Robust Recourse under Model Multiplicity." pith.science (2026). https://pith.science/paper/FC3ZZJXM

@misc{pith2026250620260,
  author       = {Pith},
  title        = {Pith review of: Argumentative Ensembling for Robust Recourse under Model Multiplicity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FC3ZZJXM}},
  note         = {Machine review of arXiv:2506.20260}
}
read the original abstract

In machine learning, it is common to obtain multiple equally performing models for the same prediction task, e.g., when training neural networks with different random seeds. Model multiplicity (MM) is the situation which arises when these competing models differ in their predictions for the same input, for which ensembling is often employed to determine an aggregation of the outputs. Providing recourse recommendations via counterfactual explanations (CEs) under MM thus becomes complex, since the CE may not be valid across all models, i.e., the CEs are not robust under MM. In this work, we formalise the problem of providing recourse under MM, which we name recourse-aware ensembling (RAE). We propose the idea that under MM, CEs for each individual model should be considered alongside their predictions so that the aggregated prediction and recourse are decided in tandem. Centred around this intuition, we introduce six desirable properties for solutions to this problem. For solving RAE, we propose a novel argumentative ensembling method which guarantees the robustness of CEs under MM. Specifically, our method leverages computational argumentation to explicitly represent the conflicts between models and counterfactuals regarding prediction results and CE validity. It then uses argumentation semantics to resolve the conflicts and obtain the final solution, in a manner which is parametric to the chosen semantics. Our method also allows for the specification of preferences over the models under MM, allowing further customisation of the ensemble. In a comprehensive theoretical analysis, we characterise the behaviour of argumentative ensembling with four different argumentation semantics. We then empirically demonstrate the effectiveness of our approach in satisfying desirable properties with eight instantiations of our method. (Abstract is shortened for arXiv.)

Figures

Figures reproduced from arXiv: 2506.20260 by the authors.

Figure 1
Figure 1. An example where a bank employs three ML models which reject a loan appli [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. AAF for the loan example where model’s predictions for the input x are given as superscripts, i.e., M1(x) = 0, M2(x) = 0 and M3(x) = 1; reciprocal attacks are represented by double-headed red arrows. Similarly, a bipolar argumentation framework (BAF) [19] is a tuple ⟨X,A,S⟩, where ⟨X,A⟩ is an AAF and S ⊆X×X is a directed relation of direct support. Given a BAF ⟨X,A,S⟩, for any α1 ∈X, we refer to S(α1)={α2∣(α2, α1) ∈… view at source ↗
Figure 3
Figure 3. An example BAF extending the previous AAF example; reciprocal supports are represented by dual-headed green arrows labelled with + and reciprocal attacks are represented by dual-headed red arrows labelled with −. We will also use notions of acceptability of sets of arguments in BAFs [19]. A set of arguments (or extension) X ⊆ X is said to set-attack any α1 ∈ X iff there exists an attack (whether direct, indirect or … view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: BAF for Example 3 where: models’ predictions for the input [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]
Figure 5
Figure 5. Figure 5: Two illustrative examples. A = {(c3,M1), (M1, c3), (c3,M2), (M2, c3)}; and S = {(M1, c1), (M2, c2), (M3, c3), (c1,M1), (c2,M2), (c3,M3)}. See Figure 5b for a graphical illus￾tration. Example 4 Example 5 stable S a,stable = {M1,M2, c1, c2}, S a,stable = {M1,M2, c1, c2},…
Figure 6
Figure 6. Figure 6: AAF for Example 6 where: models’ predictions for the input x are given as superscripts, e.g., M1(x) = 0 and M4(x) = 1; standard (reciprocal) attacks are represented by single-headed (double-headed, respectively) red arrows. Example 6 ( [PITH_FULL_IMAGE:figures/full_fi…
Figure 7
Figure 7. Figure 7: Satisfaction of desirable properties on heloc dataset. The three subplots show results respectively for the two baselines, d-preferred argumentative ensembling, and s￾preferred argumentative ensembling. Property satisfaction rates for the three datasets are visualised …
Figure 8
Figure 8. Figure 8: Satisfaction of desirable properties on compas dataset. The three subplots show results respectively for the two baselines, d-preferred argumentative ensembling, and s￾preferred argumentative ensembling. counterfactual validity, indicating that for most of the time, th…
Figure 9
Figure 9. Figure 9: Satisfaction of desirable properties on credit dataset. The three subplots show results respectively for the two baselines, d-preferred argumentative ensembling, and s￾preferred argumentative ensembling. of CEs found would drop. Argumentative ensembling, on the other h…
Figure 10
Figure 10. Figure 10: Average computation time of solving argumentative ensembling for each input. We further experiment with ∣M∣ = 50, 75, 100 to illustrate the computa￾tion time required to solve the BAF in argumentative ensembling, although these numbers of models are less likely to occ…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

82 extracted references · 74 canonical work pages

  1. [19]

    Cayrol, M

    C. Cayrol, M. Lagasquie-Schiex, On the acceptability of arguments in bipolar argumentation frameworks, in: ECSQARU 2005, 2005, pp. 378– 389

    work page 2005

  2. [1]

    Black, M

    E. Black, M. Raghavan, S. Barocas, Model multiplicity: Opportunities, concerns, and solutions, in: F AccT 2022, 2022, pp. 850–863

    2022

  3. [2]

    C. T. Marx, F. P. Calmon, B. Ustun, Predictive multiplicity in classifi- cation, in: ICML 2020, 2020, pp. 6765–6774

    2020

  4. [3]

    Breiman, Statistical modeling: The two cultures (with comments and a rejoinder by the author), Statistical science 16 (2001) 199–231

    L. Breiman, Statistical modeling: The two cultures (with comments and a rejoinder by the author), Statistical science 16 (2001) 199–231

    2001

  5. [4]

    Ganesh, A

    P. Ganesh, A. Taik, G. Farnadi, The curious case of arbitrariness in machine learning, arXiv preprint arXiv:2501.14959 (2025)

    arXiv 2025

  6. [5]

    Semenova, C

    L. Semenova, C. Rudin, R. Parr, On the existence of simpler machine learning models, in: F AccT 2022, 2022, pp. 1827–1858

    work page 2022

  7. [6]

    H. Hsu, F. P. Calmon, Rashomon capacity: A metric for predictive multiplicity in classification, in: NeurIPS 2023, 2022, pp. 28988–29000

    work page 2023

  8. [7]

    Watson-Daniels, D

    J. Watson-Daniels, D. C. Parkes, B. Ustun, Predictive multiplicity in probabilistic classification, in: AAAI 2023, 2023, pp. 10306–10314

    work page 2023

Show all 82 references
  1. [8]

    Black, K

    E. Black, K. Leino, M. Fredrikson, Selective ensembles for consistent predictions, in: ICLR 2022, 2022

    2022

  2. [9]

    Guidotti, Counterfactual explanations and how to find them: litera- ture review and benchmarking, Data Mining and Knowledge Discovery (2022) 1–55

    R. Guidotti, Counterfactual explanations and how to find them: litera- ture review and benchmarking, Data Mining and Knowledge Discovery (2022) 1–55

    2022

  3. [10]

    Karimi, G

    A. Karimi, G. Barthe, B. Sch¨ olkopf, I. Valera, A survey of algorithmic recourse: Contrastive explanations and consequential recommendations, ACM Comput. Surv. 55 (2023) 95:1–95:29

    2023

  4. [11]

    Jiang, F

    J. Jiang, F. Leofante, A. Rago, F. Toni, Robust counterfactual expla- nations in machine learning: A survey, in: Proceedings of the Thirty- Third International Joint Conference on Artificial Intelligence, IJCAI 2024, 2024, pp. 8086–8094

    2024

  5. [12]

    Pawelczyk, K

    M. Pawelczyk, K. Broelemann, G. Kasneci, On counterfactual explana- tions under predictive multiplicity, in: UAI 2020, 2020, pp. 809–818. 39

    2020

  6. [13]

    Leofante, E

    F. Leofante, E. Botoeva, V. Rajani, Counterfactual explanations and model multiplicity: a relational verification view, in: KR 2023, 2023, pp. 763–768

    2023

  7. [14]

    Coston, A

    A. Coston, A. Rambachan, A. Chouldechova, Characterizing fairness over the set of good models under selective labels, in: ICML 2021, 2021, pp. 2144–2155

    2021

  8. [15]

    Rudin, Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead, Nat

    C. Rudin, Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead, Nat. Mach. Intell. 1 (2019) 206–215

    2019

  9. [16]

    D’Amour, K

    A. D’Amour, K. Heller, D. Moldovan, B. Adlam, B. Alipanahi, A. Beu- tel, C. Chen, J. Deaton, J. Eisenstein, M. D. Hoffman, et al., Underspec- ification presents challenges for credibility in modern machine learning, JMLR 23 (2022) 10237–10297

    2022

  10. [17]

    Atkinson, P

    K. Atkinson, P. Baroni, M. Giacomin, A. Hunter, H. Prakken, C. Reed, G. R. Simari, M. Thimm, S. Villata, Towards artificial argumentation, AI Magazine 38 (2017) 25–36

    2017

  11. [18]

    Handbook of Formal Argumentation, 2018

    2018

  12. [20]

    Jiang, F

    J. Jiang, F. Leofante, A. Rago, F. Toni, Recourse under model multi- plicity via argumentative ensembling, in: Proceedings of the 23rd Inter- national Conference on Autonomous Agents and Multiagent Systems, AAMAS 2024, 2024, pp. 954–963

    2024

  13. [21]

    P. M. Dung, On the acceptability of arguments and its fundamental role in nonmonotonic reasoning, logic programming and n-person games, Artif. Intell. 77 (1995) 321–358

    1995

  14. [22]

    M. L. Wick, S. Panda, J. Tristan, Unlocking fairness: a trade-off revis- ited, in: NeurIPS 2019, 2019, pp. 8780–8789

    2019

  15. [23]

    Dutta, D

    S. Dutta, D. Wei, H. Yueksel, P. Chen, S. Liu, K. R. Varshney, Is there a trade-off between fairness and accuracy? A perspective using mismatched hypothesis testing, in: ICML 2020, 2020, pp. 2803–2813. 40

    2020

  16. [24]

    K. T. Rodolfa, H. Lamba, R. Ghani, Empirical observation of negligible fairness-accuracy trade-offs in machine learning for public policy, Nat. Mach. Intell. 3 (2021) 896–904

    2021

  17. [25]

    C. Chen, K. Lin, C. Rudin, Y. Shaposhnik, S. Wang, T. Wang, An interpretable model with globally consistent explanations for credit risk, CoRR abs/1811.12615 (2018). arXiv:1811.12615

    2018 arXiv

  18. [26]

    J. Dong, C. Rudin, Variable importance clouds: A way to explore variable importance for the set of good models, CoRR abs/1901.03209 (2019). arXiv:1901.03209

    2019 arXiv

  19. [27]

    Fisher, C

    A. Fisher, C. Rudin, F. Dominici, All models are wrong, but many are useful: Learning a variable’s importance by studying an entire class of prediction models simultaneously, J. Mach. Learn. Res. 20 (2019) 177:1–177:81

    2019

  20. [28]

    Mehrer, C

    J. Mehrer, C. J. Spoerer, N. Kriegeskorte, T. C. Kietzmann, Individual differences among deep neural network models, Nature communications 11 (2020) 5725

    2020

  21. [29]

    Black, Z

    E. Black, Z. Wang, M. Fredrikson, Consistent counterfactuals for deep models, in: ICLR 2022, 2022

    2022

  22. [30]

    D. Ley, L. Tang, M. Nazari, H. Lin, S. Srinivas, H. Lakkaraju, Consistent explanations in the face of model indeterminacy via ensembling, CoRR abs/2306.06193 (2023). arXiv:2306.06193

    2023 arXiv

  23. [31]

    C. Marx, Y. Park, H. Hasson, Y. Wang, S. Ermon, L. Huan, But are you sure? an uncertainty-aware perspective on explainable AI, in: AISTATS 2023, 2023, pp. 7375–7391

    2023

  24. [32]

    R. Xin, C. Zhong, Z. Chen, T. Takagi, M. I. Seltzer, C. Rudin, Exploring the whole rashomon set of sparse decision trees, in: NeurIPS 2022, 2022, pp. 14071–14084

    2022

  25. [33]

    Zhong, Z

    C. Zhong, Z. Chen, J. Liu, M. I. Seltzer, C. Rudin, Exploring and interacting with the set of good sparse generalized additive models, in: NeurIPS 2023, 2023. 41

    2023

  26. [34]

    H. Hsu, G. Li, S. Hu, C.-F. Chen, Dropout-based rashomon set explo- ration for efficient predictive multiplicity estimation, in: The Twelfth International Conference on Learning Representations, 2024

    2024

  27. [35]

    H. Hsu, I. Brugere, S. Sharma, F. L´ ecu´ e, R. Chen, Rashomongb: An- alyzing the rashomon effect and mitigating predictive multiplicity in gradient boosting, in: NeurIPS 2024, 2024

    2024

  28. [36]

    A. Roth, A. Tolbert, S. Weinstein, Reconciling individual probability forecasts, in: F AccT 2023, 2023, pp. 101–110

    2023

  29. [37]

    Cavus, P

    M. Cavus, P. Biecek, An experimental study on the rashomon effect of balancing methods in imbalanced classification, CoRR abs/2405.01557 (2024). arXiv:2405.01557

    2024 arXiv

  30. [38]

    Hamman, P

    F. Hamman, P. Dissanayake, S. Mishra, F. Lecue, S. Dutta, Quantifying prediction consistency under model multiplicity in tabular llms, arXiv preprint arXiv:2407.04173 (2024)

    2024 arXiv

  31. [39]

    Potyka, Y

    N. Potyka, Y. Zhu, Y. He, E. Kharlamov, S. Staab, Robust knowl- edge extraction from large language models using social choice theory, in: Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2024, 2024, pp. 1593–1601

    2024

  32. [40]

    Y. Zhu, N. Potyka, M. Nayyeri, B. Xiong, Y. He, E. Kharlamov, S. Staab, Predictive multiplicity of knowledge graph embeddings in link prediction, in: Findings of the Association for Computational Lin- guistics: EMNLP 2024, 2024, pp. 334–354

    2024

  33. [41]

    Tolomei, F

    G. Tolomei, F. Silvestri, A. Haines, M. Lalmas, Interpretable predictions of tree-based ensembles via actionable feature tweaking, in: KDD 2017, 2017, pp. 465–474

    2017

  34. [42]

    Wachter, B

    S. Wachter, B. D. Mittelstadt, C. Russell, Counterfactual explanations without opening the black box: Automated decisions and the GDPR, Harv. JL & Tech. 31 (2017) 841

    2017

  35. [43]

    Ustun, A

    B. Ustun, A. Spangher, Y. Liu, Actionable recourse in linear classifica- tion, in: F AT 2019, 2019, pp. 10–19. 42

    2019

  36. [44]

    Dhurandhar, P

    A. Dhurandhar, P. Chen, R. Luss, C. Tu, P. Ting, K. Shanmugam, P. Das, Explanations based on the missing: Towards contrastive expla- nations with pertinent negatives, in: NeurIPS 2018, 2018, pp. 590–601

    2018

  37. [45]

    R. K. Mothilal, A. Sharma, C. Tan, Explaining machine learning clas- sifiers through diverse counterfactual explanations, in: F AT 2020, 2020, pp. 607–617

    2020

  38. [46]

    A. V. Looveren, J. Klaise, Interpretable counterfactual explanations guided by prototypes, in: ECML PKDD 2021, volume 12976 of Lecture Notes in Computer Science , 2021, pp. 650–665

    2021

  39. [47]

    E. M. Kenny, M. T. Keane, On generating plausible counterfactual and semi-factual explanations for deep learning, in: AAAI 2021, 2021, pp. 11575–11585

    2021

  40. [48]

    T. Wu, M. T. Ribeiro, J. Heer, D. S. Weld, Polyjuice: Generating counterfactuals for explaining, evaluating, and improving models, in: ACL/IJCNLP 2021, 2021, pp. 6707–6723

    2021

  41. [49]

    Jiang, T

    J. Jiang, T. Bewley, S. Mishra, F. Lecue, M. Veloso, Interpreting lan- guage reward models via contrastive explanations, in: The 13th Inter- national Conference on Learning Representations, ICLR, 2025

    2025

  42. [50]

    Upadhyay, S

    S. Upadhyay, S. Joshi, H. Lakkaraju, Towards robust and reliable algo- rithmic recourse, in: NeurIPS 2021, 2021, pp. 16926–16937

    2021

  43. [51]

    Dutta, J

    S. Dutta, J. Long, S. Mishra, C. Tilli, D. Magazzeni, Robust counter- factual explanations for tree-based ensembles, in: ICML 2022, 2022, pp. 5742–5756

    2022

  44. [52]

    Jiang, F

    J. Jiang, F. Leofante, A. Rago, F. Toni, Formalising the robustness of counterfactual explanations for neural networks, in: AAAI 2023, 2023, pp. 14901–14909

    2023

  45. [53]

    Hamman, E

    F. Hamman, E. Noorani, S. Mishra, D. Magazzeni, S. Dutta, Robust counterfactual explanations for neural networks with probabilistic guar- antees, in: ICML 2023, 2023, pp. 12351–12367. 43

    2023

  46. [54]

    Jiang, J

    J. Jiang, J. Lan, F. Leofante, A. Rago, F. Toni, Provably robust and plausible counterfactual explanations for neural networks via robust op- timisation, in: Proceedings of the 15th Asian Conference on Machine Learning, ACML, volume 222 of PMLR, 2024, pp. 582–597

    2024

  47. [55]

    Jiang, F

    J. Jiang, F. Leofante, A. Rago, F. Toni, Interval abstractions for robust counterfactual explanations, Artif. Intell. 336 (2024) 104218

    2024

  48. [56]

    Mishra, S

    S. Mishra, S. Dutta, J. Long, D. Magazzeni, A survey on the ro- bustness of feature importance and counterfactual explanations, CoRR abs/2111.00358 (2021). arXiv:2111.00358

    2021 arXiv

  49. [57]

    Leofante, M

    F. Leofante, M. Wicker, Robustness of Counterfactual Explanations, Springer Nature Switzerland, Cham, 2025, pp. 17–40. URL: https://doi. org/10.1007/978-3-031-89022-2 3. doi: 10.1007/978-3-031-89022-2_ 3

    2025 doi

  50. [58]

    Cabrio, S

    E. Cabrio, S. Villata, A natural language bipolar argumentation ap- proach to support users in online debate interactions †, Argument Com- put. 4 (2013) 209–230

    2013

  51. [59]

    Cyras, D

    K. Cyras, D. Letsios, R. Misener, F. Toni, Argumentation for explain- able scheduling, in: AAAI 2019, 2019, pp. 2752–2759

    2019

  52. [60]

    Irwin, A

    B. Irwin, A. Rago, F. Toni, Forecasting argumentation frameworks, in: KR 2022, 2022, pp. 533–543

    2022

  53. [61]

    Cyras, A

    K. Cyras, A. Rago, E. Albini, P. Baroni, F. Toni, Argumentative XAI: A survey, in: IJCAI 2021, 2021, pp. 4392–4399

    2021

  54. [62]

    Vassiliades, N

    A. Vassiliades, N. Bassiliades, T. Patkos, Argumentation and explain- able artificial intelligence: a survey, Knowl. Eng. Rev. 36 (2021) e5

    2021

  55. [63]

    Y. Guo, T. Yu, L. Bai, J. Tang, Y. Ruan, Y. Zhou, Argumentative explanation for deep learning: A survey, in: ICUS 2023, 2023, pp. 1738–1743

    2023

  56. [64]

    Potyka, Interpreting neural networks as quantitative argumentation frameworks, in: AAAI 2021, 2021, pp

    N. Potyka, Interpreting neural networks as quantitative argumentation frameworks, in: AAAI 2021, 2021, pp. 6463–6470. 44

    2021

  57. [65]

    A. Dejl, C. He, P. Mangal, H. Mohsin, B. Surdu, E. Voinea, E. Albini, P. Lertvittayakumjorn, A. Rago, F. Toni, Argflow: A toolkit for deep argumentative explanations for neural networks, in: AAMAS 2021, 2021, pp. 1761–1763

    2021

  58. [66]

    S. T. Timmer, J. C. Meyer, H. Prakken, S. Renooij, B. Verheij, Ex- plaining bayesian networks using argumentation, in: ECSQARU 2015, 2015, pp. 83–92

    2015

  59. [67]

    Potyka, X

    N. Potyka, X. Yin, F. Toni, Explaining random forests using bipolar argumentation and markov networks, in: AAAI 2023, 2023, pp. 9453– 9460

    2023

  60. [68]

    Leofante, H

    F. Leofante, H. Ayoobi, A. Dejl, G. Freedman, D. Gorur, J. Jiang, G. Paulino-Passos, A. Rago, A. Rapberger, F. Russo, X. Yin, D. Zhang, F. Toni, Contestable AI needs computational argumentation, in: KR 2024, 2024

    2024

  61. [69]

    Abchiche-Mimouni, L

    N. Abchiche-Mimouni, L. Amgoud, F. Zehraoui, Explainable ensemble classification model based on argumentation, in: AAMAS 2023, 2023, pp. 2367–2369

    2023

  62. [70]

    FICO, Explainable machine learning challenge, 2018

    2018

  63. [71]

    S. M. Julia Angwin, Jeff Larson, L. Kirchner, There’s software used across the country to predict future criminals. and it’s biased against blacks., 2016

    2016

  64. [72]

    Hofmann, Statlog (German Credit Data), UCI Machine Learning Repository, 1994

    H. Hofmann, Statlog (German Credit Data), UCI Machine Learning Repository, 1994

    1994

  65. [73]

    Brughmans, P

    D. Brughmans, P. Leyman, D. Martens, NICE: an algorithm for near- est instance counterfactual explanations, Data Mining and Knowledge Discovery (2023) 1–39

    2023

  66. [74]

    Jiang, L

    J. Jiang, L. Marzari, A. Purohit, F. Leofante, Robustx: Robust counter- factual explanations made easy, arXiv preprint arXiv:2502.13751 (2025)

    2025 arXiv

  67. [75]

    S. H. Nielsen, S. Parsons, A generalization of dung’s abstract frame- work for argumentation: Arguing with sets of attacking arguments, in: ArgMAS 2006, 2006, pp. 54–73. 45

    2006

  68. [76]

    Flouris, A

    G. Flouris, A. Bikakis, A comprehensive study of argumentation frame- works with sets of attacking arguments, Int. J. Approx. Reason. 109 (2019) 55–86

    2019

  69. [77]

    Dvor´ ak, M

    W. Dvor´ ak, M. K¨ onig, M. Ulbricht, S. Woltran, Rediscovering argu- mentation principles utilizing collective attacks, in: KR 2022, 2022, pp. 122–131

    2022

  70. [78]

    Dimopoulos, W

    Y. Dimopoulos, W. Dvor´ ak, M. K¨ onig, A. Rapberger, M. Ulbricht, S. Woltran, Sets attacking sets in abstract argumentation, in: NMR 2023, 2023, pp. 22–31

    2023

  71. [79]

    Modgil, Reasoning about preferences in argumentation frameworks, Artif

    S. Modgil, Reasoning about preferences in argumentation frameworks, Artif. Intell. 173 (2009) 901–934

    2009

  72. [80]

    T. J. M. Bench-Capon, Value-based argumentation frameworks, in: NMR 2002, 2002, pp. 443–454

    2002

  73. [81]

    X. Fan, F. Toni, On computing explanations in abstract argumentation, in: ECAI 2014, 2014, pp. 1005–1006

    2014

  74. [82]

    Z. Zeng, C. Miao, C. Leung, Z. Shen, J. J. Chin, Computing argumenta- tive explanations in bipolar argumentation frameworks, in: AAAI 2019, 2019, pp. 10079–10080. 46

    2019

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.