Pith. sign in

REVIEW 35 references

CEL: Comprehensive Counterfactual Explanations Library and Benchmark

T0 review · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CEL standardizes the messy evaluation of counterfactual explanations and shows that no method wins on every metric.

arxiv 2607.22045 v2 pith:TC5RWR74 submitted 2026-07-24 cs.LG cs.AI

Oleksii Furman , {L}ukasz Lenkiewicz , Marcel Musia{l}ek , Maciej Zięba This is my paper
classification cs.LGcs.AI
keywords counterfactualexplanationsexplainableAIbenchmarkevaluationprotocolplausibilitysparsityalgorithmicrecoursereproducibility
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Counterfactual explanations tell you what input changes would flip a model's decision, but comparing methods has been inconsistent: studies differ in datasets, models, preprocessing, and metrics. The paper introduces CEL, a library and benchmark that fixes all of that with 18 datasets and 14 methods behind one protocol and one metric suite (validity, proximity, sparsity, plausibility). The authors' central claim is that under this unified setup, no single method uniformly dominates; each method wins on some dimensions and loses on others, and the trade-offs are now visible and reproducible. If true, CEL gives researchers a common yardstick, helps practitioners pick methods by their application's priorities, and makes future method development and integration easier.

What carries the argument

The load-bearing component is the CEL framework: a BaseCounterfactualMethod interface that all 14 methods implement, a config-driven data and model module fixing preprocessing and predictive backbones, and a MetricsOrchestrator that computes a fixed set of metrics for every method and dataset. Plausibility is scored by the log-likelihood of a normalizing-flow density estimator (masked autoregressive flow) trained on each class, which measures whether generated counterfactuals fall in high-density regions of the data manifold. This uniform harness ensures that observed differences are due to method choice rather than experimental configuration.

What would settle it

Take the same 14 methods and the same protocol, add per-dataset hyperparameter optimization for every method, and swap the plausibility metric from a normalizing-flow likelihood to a kernel-density estimate; if one method then dominates on all metrics or the ranking order reverses substantially, the paper's 'no uniform dominance' conclusion would be an artifact of the tuning budget or metric choice. Alternatively, an independent re-implementation of a subset of methods (e.g., PPCEF, DiCE, Wachter) under the CEL protocol that leads to a different ordering would falsify the claim that the benchm

Watch

Extended reading notes

Core claim

Using this unified protocol, the authors find that among local methods CCHVAE and PPCEF achieve near-perfect validity with high-likelihood counterfactuals, while CADEX and Wachter generate closer but less realistic ones; DiCE and CEGP are the sparsest. For global methods, GLOBE-CE beats AReS on validity, but all show high variance and lower plausibility. Group-wise GLANCE matches local-level validity with interpretable subgroup rules, while T-CREx yields closer, more plausible counterfactuals but with much lower success rates. For regression, the simple Wachter method beats CEARM on accuracy, plausibility, and speed. The paper's central claim is that under this unified protocol, no method un

Load-bearing premise

The comparison is only fair if the 14 reimplementations are faithful and equally well tuned, and if the chosen metrics are neutral across method families; any bias in tuning effort or metric selection would make the rankings reflect implementation quality rather than method quality.

Editorial extensions

If this is right

  • With one protocol across datasets, splits, models, and metrics, method rankings become stable across studies, so incremental progress in counterfactual generation can actually be measured.
  • New methods can be dropped into the common interface and compared against 14 baselines without re-implementing the evaluation pipeline.
  • The observed trade-offs give concrete guidance: where minimal feature changes matter, CADEX and Wachter are strong; where realism matters, PPCEF and CCHVAE; where sparsity matters, DiCE and CEGP.
  • The results identify specific weak spots, such as AReS's validity collapse on Adult and T-CREx's low success rate, which become targets for improvement.
  • The same infrastructure can be extended to test counterfactual explanation robustness and fairness, as the authors suggest.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the plausibility metric is a normalizing-flow log-likelihood and several methods (e.g., PPCEF, CCHVAE) natively optimize density under the same flow family, the reported plausibility rankings may partly reflect metric alignment; a benchmark that changes only the density estimator (e.g., kernel density) would show whether rankings hold.
  • The three-way trade-off data across validity, sparsity, and plausibility invites a decision-support layer that takes application-specific weights and suggests a default method — an automation the paper does not build but its data enables.
  • The success of GLANCE as a group-wise method suggests a broader design principle: intermediate-granularity rules can combine the validity of local methods with the interpretability of global ones, pointing toward hierarchical recourse systems.
  • The regression result that a simple gradient-based method beats a sophisticated Bayesian optimizer suggests that for continuous targets, search-space structure matters more than optimizer complexity; this invites stress-testing on higher-dimensional regression.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Circularity Check

2 steps flagged · score 6.0 of 10

Plausibility metric is the optimization objective of PPCEF, a benchmarked method from the same group, so the 'fair comparison' claim is partially circular.

  1. fitted input called prediction [Section 4.4 (Metrics), Section 4.3 (Methods), Section 4.6 (Results, Local Methods)]
    "Plausibility: Assesses how well counterfactuals fit the data distribution using Log-Likelihood from density estimation model. We use MAF [22] for this benchmark, due to its superiority verified by Wielopolski et al. [33]. ... PPCEF [33], a flow-based approach with an explicit probabilistic formulation. ... PPCEF and CCHVAE demonstrate the high log-density, indicating its counterfactuals align well with the data manifold."

    The benchmark's plausibility metric is the log-likelihood under a MAF density estimator. PPCEF, cited as [33], is a flow-based method whose objective is to generate counterfactuals with high likelihood under a normalizing flow. The paper justifies the MAF choice by the same PPCEF paper [33]. Therefore, PPCEF is evaluated on essentially its own training objective: its high log-density is not an independent benchmark discovery but a direct consequence of optimizing for the metric. This makes the fairness claim circular for the plausibility axis: the metric is not neutral across the method families being ranked.

  2. self citation load bearing [Section 4.2 (Models) and Section 4.4 (Metrics)]
    "To support plausibility evaluation and methods that rely on density estimation we utilise Masked Autoregressive Flow, accordingly to Wielopolski et al. [33]. ... We use MAF [22] for this benchmark, due to its superiority verified by Wielopolski et al. [33]."

    The selection of the central plausibility metric is justified solely by citing [33], the authors' own prior work (Wielopolski, Furman, Stefanowski, Zięba). That same prior work introduces PPCEF, a benchmarked method that scores highly on this metric. No independent external validation of MAF's 'superiority' as a neutral evaluation metric is provided; the load-bearing validation comes from the same author group that contributes a top-scoring method, so the benchmark's fairness premise rests partly on a self-citation chain.

full rationale

CEL's central contribution is a unified, reproducible benchmark, and most of the protocol (datasets, backbones, method reimplementations, validity/proximity/sparsity metrics) is independent and externally checkable. The main circularity is concentrated in the plausibility axis: the log-density metric is defined via MAF, and the MAF choice is justified by the authors' own PPCEF paper [33], while PPCEF's optimization objective is high likelihood under a normalizing flow. Thus the finding that 'PPCEF demonstrates the high log-density' is partly by construction rather than by independent evaluation. The self-citation of [33] is load-bearing for metric selection. However, the benchmark also reports other metrics not aligned with PPCEF's objective, and the overall library contribution does not reduce to this metric, so the circularity is partial, not total. Separately, LOF/IsoForest are mentioned in the abstract and architecture but not defined in Section 4.4; that is a rigor/reproducibility concern, not a derivational circularity, and does not affect this score.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim does not rest on a mathematical derivation, so the ledger captures the benchmark-design choices that the rankings depend on. Most are reasonable but unverified; the plausibility-metric choice is the one with real potential to bias the comparison toward flow-based methods.

free parameters (2)
  • Regression target shift = 20% of target range
    Chosen by hand to define the regression counterfactual target; affects all reported MAE values.
  • Per-method hyperparameters = not reported
    Rankings depend on tuning of the 14 reimplemented methods, but no hyperparameter details are given in the paper.
assumptions (4)
  • domain assumption Faithful reimplementation of each cited method
    The benchmark's rankings are only meaningful if each of the 14 methods is correctly reimplemented and tuned; the paper does not provide evidence for this.
  • domain assumption MAF log-likelihood is a valid proxy for counterfactual plausibility
    Section 4.4 defines plausibility solely by log-density under a MAF; if this metric is not neutral or valid, the plausibility rankings are biased.
  • domain assumption Representative selection of datasets and binary class-pair reduction preserves conclusions
    Section 4.5 restricts multiclass datasets to a selected class pair and the paper reports only a representative subset of the full benchmark; this assumes the omitted settings would not change qualitative conclusions.
  • domain assumption 5-fold cross-validation gives stable rankings
    Used in Section 4.5; no significance testing is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CEL: Comprehensive Counterfactual Explanations Library and Benchmark." pith.science (2026). https://pith.science/paper/TC5RWR74

@misc{pith2026260722045,
  author       = {Pith},
  title        = {Pith review of: CEL: Comprehensive Counterfactual Explanations Library and Benchmark},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TC5RWR74}},
  note         = {Machine review of arXiv:2607.22045}
}
read the original abstract

Counterfactual explanations are a prominent approach in explainable artificial intelligence (xAI), providing actionable guidance on what input changes would alter a model's prediction to a desired outcome. While early methods primarily focused on minimal feature changes, recent work incorporates additional properties such as sparsity, actionability and plausibility. Despite this progress, fair and systematic evaluation remains challenging. Existing studies often rely on different data splits, predictive models, and evaluation metrics, which limits objective comparison across methods. To fill this gap, we introduce CEL (Counterfactual Explanations Library), a unified library and benchmark for counterfactual explanations designed to support consistent implementation and evaluation. CEL includes 18 datasets of varying size and complexity and provides implementations or reimplementations of 14 widely used counterfactual methods. Using this standardized setup, we conduct a comprehensive quantitative comparison across a variety of methods on datasets that differ in size, number, and types of attributes. The evaluation protocol incorporates multiple complementary metrics capturing validity, coverage, sparsity, proximity, and distributional plausibility, including density- and outlier-based measures to assess the realism of generated counterfactuals. To the best of our knowledge, this is the first comprehensive benchmark that systematically evaluates recent counterfactual explanation methods within a unified and reproducible framework. While prior libraries and benchmarking efforts exist in the literature, many are outdated, limited in scope, or lack consistent evaluation protocols. The proposed benchmark aims to improve reproducibility, enable fair comparison, and establish a workbench for the development of future counterfactual explanation methods.

Figures

Figures reproduced from arXiv: 2607.22045 by the authors.

Figure 1
Figure 1. Overview of the CEL architecture. It consists of four interconnected modules: (i) a Data Module handling dataset loading, preprocessing, and constraint specifica￾tion; (ii) a Model Module providing predictive and probabilistic backbones; (iii) an Explanation Engine supporting local, global, and group-wise counterfactual methods; and (iv) a Metrics Orchestrator computing validity, proximity, sparsity, and plausibilit… view at source ↗
Figure 2
Figure 2. Performance of local counterfactual methods across seven classification datasets. Rows represent datasets and columns represent evaluation metrics (Valid￾ity, Euclidean-Hamming Distance, Sparsity, Log-Density, Computation Time). Each boxplot shows the distribution of metric values across methods: Artelt (blue), DiCE (light blue), CCHVAE (orange), PPCEF (light orange), CEGP (green), CADEX (red), and SACE (pink). 4.6 … view at source ↗
Figure 3
Figure 3. Performance of counterfactual methods on regression tasks. Rows represent regression datasets and columns represent metrics (MAE, Sparsity, Log-Density, Com￾putation Time). Methods: CEARM (blue), WACH (orange). Local Methods [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Performance of global counterfactual methods. Rows represent datasets and columns represent metrics. Methods: AReS (pink), GLOBE-CE (dark brown), Glob￾alGLANCE (brown). Global Methods Global methods provide dataset-level insights by learning transformations applicable …
Figure 5
Figure 5. Figure 5: Performance of group-wise counterfactual methods. Rows represent datasets and columns represent metrics. Methods: GLANCE (dark pink), T-CREx (light pink). Group-wise Methods Group-wise methods bridge the gap between local and global approaches by generating explanation…

Review history (2 revisions) →

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

35 extracted references · 4 linked inside Pith

  1. [1]

    In: International conference on artificial neural networks

    Artelt, A., Hammer, B.: Convex density constraints for computing plausible coun- terfactual explanations. In: International conference on artificial neural networks. pp. 353–365. Springer (2020)

    2020

  2. [2]

    UCI Machine Learning Repository (1996)

    Becker, B., Kohavi, R.: Adult Census Income. UCI Machine Learning Repository (1996). https://doi.org/10.24432/C5XW20 CEL: Counterfactual Explanations Library and Benchmark 15

    doi:10.24432/c5xw20 1996

  3. [3]

    In: Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024

    Bewley, T., Amoukou, S.I., Mishra, S., Magazzeni, D., Veloso, M.: Counterfactual metarules for local and global recourse. In: Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net (2024),https://openreview.net/forum?id=Ad9msn1SKC

    2024

  4. [4]

    In: Advances in Neural Information Processing Systems 31: Annual Con- ference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada

    Dhurandhar, A., Chen, P., Luss, R., Tu, C., Ting, P., Shanmugam, K., Das, P.: Ex- planations based on the missing: Towards contrastive explanations with pertinent negatives. In: Advances in Neural Information Processing Systems 31: Annual Con- ference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada. pp. 59...

    2018

  5. [5]

    The Annals of Statistics32(01 2004)

    Efron, B., Hastie, T., Johnstone, I., Tibshirani, R.: Least angle regression” (with discussions). The Annals of Statistics32(01 2004)

    2004

  6. [6]

    com/s/explainable-machine-learning-challenge(2018)

    FICO: FICO Explainable Machine Learning Challenge.https://community.fico. com/s/explainable-machine-learning-challenge(2018)

    2018

  7. [7]

    UCI Machine Learning Repository (1981), DOI: https://doi.org/10.24432/C5XG7R

    Gerritsma, J., Onnink, R., Versluis, A.: Yacht Hydrodynamics. UCI Machine Learning Repository (1981), DOI: https://doi.org/10.24432/C5XG7R

    doi:10.24432/c5xg7r 1981

  8. [8]

    Data Mining and Knowledge Discovery38(5), 2770–2824 (2024)

    Guidotti, R.: Counterfactual explanations and how to find them: literature re- view and benchmarking. Data Mining and Knowledge Discovery38(5), 2770–2824 (2024)

    2024

Show all 35 references
  1. [9]

    UCI Machine Learning Repository (1994), DOI: https://doi.org/10.24432/C5NC77

    Hofmann, H.: Statlog (German Credit Data). UCI Machine Learning Repository (1994), DOI: https://doi.org/10.24432/C5NC77

    1994 doi

  2. [10]

    Financial Man- agement48(4), 1009–1029 (2019)

    Jagtiani, J., Lemieux, C.: The roles of alternative data and machine learning in fintech lending: evidence from the lendingclub consumer platform. Financial Man- agement48(4), 1009–1029 (2019)

    2019

  3. [11]

    arXiv preprint arXiv:2502.13751 (2025)

    Jiang, J., Marzari, L., Purohit, A., Leofante, F.: Robustx: Robust counterfactual explanations made easy. arXiv preprint arXiv:2502.13751 (2025)

    2025 arXiv

  4. [12]

    In: International conference on artificial intelligence and statistics

    Karimi, A.H., Barthe, G., Balle, B., Valera, I.: Model-agnostic counterfactual ex- planations for consequential decisions. In: International conference on artificial intelligence and statistics. pp. 895–905. PMLR (2020)

    2020

  5. [13]

    arXiv preprint arXiv:2405.18921 (2024)

    Kavouras, L., Psaroudaki, E., Tsopelas, K., Rontogiannis, D., Theologitis, N., Sacharidis, D., Giannopoulos, G., Tomaras, D., Markou, K., Gunopulos, D., et al.: Glance: Global actions in a nutshell for counterfactual explainability. arXiv preprint arXiv:2405.18921 (2024)

    2024 arXiv

  6. [14]

    In: Inter- national Conference on Case-Based Reasoning

    Keane, M.T., Smyth, B.: Good counterfactuals and where to find them: A case- based technique for generating counterfactuals for explainable ai (xai). In: Inter- national Conference on Case-Based Reasoning. pp. 163–178. Springer (2020)

    2020

  7. [15]

    In: International conference on machine learn- ing

    Ley, D., Mishra, S., Magazzeni, D.: Globe-ce: A translation based approach for global counterfactual explanations. In: International conference on machine learn- ing. pp. 19315–19342. PMLR (2023)

    2023

  8. [16]

    Lecture Notes in Computer Science, vol

    Looveren, A.V., Klaise, J.: Interpretable counterfactual explanations guided by prototypes.In:MachineLearningandKnowledgeDiscoveryinDatabases.Research Track - European Conference, ECML PKDD 2021, Bilbao, Spain, September 13- 17, 2021, Proceedings, Part II. Lecture Notes in Com...

    2021

  9. [17]

    Kaggle,https://www

    (LSAC), L.S.A.C.: Law School Admissions Bar Passage. Kaggle,https://www. kaggle.com/datasets/danofer/law-school-admissions-bar-passage

  10. [18]

    In: Pacific Rim international conference on artificial intelligence

    Moore, J., Hammerla, N., Watkins, C.: Explaining deep learning models with con- strained adversarial examples. In: Pacific Rim international conference on artificial intelligence. pp. 43–56. Springer (2019)

    2019

  11. [19]

    Decision Support Systems62, 22–31 (2014) 16 O

    Moro, S., Cortez, P., Rita, P.: A data-driven approach to predict the success of bank telemarketing. Decision Support Systems62, 22–31 (2014) 16 O. Furman et al

    2014

  12. [20]

    In: Proceedings of the 2020 conference on fairness, accountability, and transparency

    Mothilal, R.K., Sharma, A., Tan, C.: Explaining machine learning classifiers through diverse counterfactual explanations. In: Proceedings of the 2020 conference on fairness, accountability, and transparency. pp. 607–617 (2020)

    2020

  13. [21]

    Applied Sciences11(16), 7274 (2021)

    de Oliveira, R.M.B., Martens, D.: A framework and benchmarking study for coun- terfactual generating methods on tabular data. Applied Sciences11(16), 7274 (2021)

    2021

  14. [22]

    Advances in neural information processing systems30(2017)

    Papamakarios, G., Pavlakou, T., Murray, I.: Masked autoregressive flow for density estimation. Advances in neural information processing systems30(2017)

    2017

  15. [23]

    arXiv preprint arXiv:2108.00783 (2021)

    Pawelczyk, M., Bielawski, S., Heuvel, J.v.d., Richter, T., Kasneci, G.: Carla: a python library to benchmark algorithmic recourse and counterfactual explanation algorithms. arXiv preprint arXiv:2108.00783 (2021)

    2021 arXiv

  16. [24]

    In: Proceedings of the web conference 2020

    Pawelczyk, M., Broelemann, K., Kasneci, G.: Learning model-agnostic counterfac- tual explanations for tabular data. In: Proceedings of the web conference 2020. pp. 3126–3132 (2020)

    2020

  17. [25]

    Journal of ma- chine learning research22(164), 1–20 (2021)

    Pineau, J., Vincent-Lamarre, P., Sinha, K., Larivière, V., Beygelzimer, A., d’Alché Buc, F., Fox, E., Larochelle, H.: Improving reproducibility in machine learning research (a report from the neurips 2019 reproducibility program). Journal of ma- chine learning research22(164),...

    2019

  18. [26]

    Advances in Neural Information Pro- cessing Systems33, 12187–12198 (2020)

    Rawal, K., Lakkaraju, H.: Beyond individualized recourse: Interpretable and in- teractive summaries of actionable recourses. Advances in Neural Information Pro- cessing Systems33, 12187–12198 (2020)

    2020

  19. [27]

    Repository, U.M.L.: Wine Data Set.https://archive.ics.uci.edu/dataset/ 109/wine(1991)

    1991

  20. [28]

    ics.uci.edu/dataset/80/optical+recognition+of+handwritten+digits(1998)

    Repository,U.M.L.:OpticalRecognitionofHandwrittenDigits.https://archive. ics.uci.edu/dataset/80/optical+recognition+of+handwritten+digits(1998)

    1998

  21. [29]

    Repository, U.M.L.: Audit Data Set.https://archive.ics.uci.edu/dataset/ 475/audit+data(2016)

    2016

  22. [30]

    arXiv preprint arXiv:2106.15212 (2021)

    Spooner, T., Dervovic, D., Long, J., Shepard, J., Chen, J., Magazzeni, D.: Counterfactual explanations for arbitrary regression models. arXiv preprint arXiv:2106.15212 (2021)

    2021 arXiv

  23. [31]

    Machine Learning 104(1), 55–98 (July 2016)

    Spyromitros-Xioufis, E., Tsoumakas, G., Groves, W., Vlahavas, I.: Multi-target regression via input space expansion: treating targets as inputs. Machine Learning 104(1), 55–98 (July 2016). https://doi.org/10.1007/s10994-016-5546-z,https:// doi.org/10.1007/s10994-016-5546-z

    2016 doi

  24. [32]

    Wachter, S., Mittelstadt, B., Russell, C.: Counterfactual explanations without opening the black box: Automated decisions and the gdpr. Harv. JL & Tech.31, 841 (2017)

    2017

  25. [33]

    arXiv preprint arXiv:2405.17640 (2024)

    Wielopolski, P., Furman, O., Stefanowski, J., Zięba, M.: Probabilistically plausible counterfactual explanations with normalizing flows. arXiv preprint arXiv:2405.17640 (2024)

    2024

  26. [34]

    UCI Machine Learning Repository (1998), DOI: https://doi.org/10.24432/C5PK67

    Yeh, I.C.: Concrete Compressive Strength. UCI Machine Learning Repository (1998), DOI: https://doi.org/10.24432/C5PK67

    1998 doi

  27. [35]

    UCI Machine Learning Repository (2009), DOI: https://doi.org/10.24432/C55S3H

    Yeh, I.C.: Default of Credit Card Clients. UCI Machine Learning Repository (2009), DOI: https://doi.org/10.24432/C55S3H

    2009 doi

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.