Pith. sign in

REVIEW 3 major objections 4 minor 29 references

Co-constructing Explanations for AI Systems using Provenance

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that AI explanations should be co-constructed in dialogue, anchored by data provenance so they stay faithful to how the system actually produced its result.

desk verdict Honest vision paper with a working prototype, but the evaluation framework never actually checks the 'grounded in provenance' half of the central claim. read the letter →

arxiv 2507.17761 v1 pith:YOUJTJ3O submitted 2025-05-31 cs.HC

Jan-Christoph Kalo , Fina Polat , Shubha Guha , Paul Groth This is my paper
classification cs.HC
keywords co-constructionofexplanationsprovenanceexplainableAILLM-as-a-judgeusersimulationdialogueevaluationtrace-basedinteractiveexplanationagent
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that explanations of complex AI systems should be co-constructed in dialogue rather than delivered as a single output, and that data provenance should anchor that dialogue so the resulting explanation is both tailored to the user and faithful to how the system actually produced its result. To make the idea concrete, the authors built a prototype chatbot that explains the outputs of a neural class expression learner by feeding provenance into an LLM prompt and letting users ask follow-up questions. They also propose a scalable evaluation framework in which LLM-simulated user personas chat with the explainer and an LLM judge scores the traces. If the approach works, provenance-based explanations could be adapted to many user types and many machine-learning tasks without expensive human evaluation. The paper is a vision-and-prototype contribution: it establishes a scenario and a measurement method, not a field-tested system.

What carries the argument

The central object is the ENEXA explanation chatbot's dialog engine, which mediates between the user and an LLM by maintaining chat history and injecting provenance metadata, namely the underlying data sources and the information extraction procedure used to learn the class expression, into the prompt. This lets the LLM mix provenance with its own background knowledge and tailor the level of detail to the user. The evaluation machinery is the automated framework that pairs LLM-simulated user personas with a separate LLM judge scoring seven criteria (clarity, depth, correctness and fidelity, relevance and focus, persona appropriateness, transparency, and engagement) on a five-point scale.

What would settle it

Run a validation study where the same chat traces are scored by both the LLM judge and a panel of human raters matched to the six persona profiles, then check the correlation on each criterion. Alternatively, deploy the chatbot with real users in those roles and measure whether the explanations improve task success or user understanding; low correlation or poor real-user performance would undercut the central claim.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that provenance can be the grounding for an interactive co-construction process: the user and an explanation agent together shape what is explained, while the underlying workflow trace keeps the explanation accurate. The authors demonstrate this with ENEXA, a chatbot that verbalizes class expressions learned from positive and negative examples, injects provenance about data sources and the information extraction procedure into the LLM prompt, and answers follow-up questions so the explanation narrows onto what the user actually needs. They report that in automated evaluation over six personas and ten class expressions, the explainer scored well on correctness and fidelity and on relevance and focus, with more mixed results on engagement and depth. The paper's second claim is that the combination of LLM user simulation and an LLM-as-a-judge offers a route to large-scale, reproducible evaluation of such co-constructive explanation systems.

Load-bearing premise

The evaluation framework's scores are assumed to stand in for real human judgments; if LLM-simulated personas and the LLM judge do not reflect actual users, the reported successes do not show that co-construction works for humans.

Editorial extensions

If this is right

  • If the central claim is correct, provenance-based explanations can be personalized in dialogue without sacrificing faithfulness to the underlying computation.
  • The evaluation framework makes it practical to compare explanation strategies across many personas and many tasks at scale.
  • The approach extends trace-based explanation to interactive settings, letting developers and product owners interrogate the entire data-to-result pipeline rather than just a model's features.
  • The seven scoring criteria offer a concrete template for measuring the quality of co-constructed explanations.
  • The identified weaknesses in engagement and depth point to longer dialogues and richer provenance as the immediate next improvements.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to validate the LLM judge against human raters on the same chat traces; if agreement is high, the framework becomes a practical screening tool for explanation design.
  • The persona set could grow into a reusable battery of hundreds of personas, as the paper's vision section anticipates, enabling standardized comparison across different explanation agents.
  • The approach likely transfers beyond class expression learning to any pipeline with recorded provenance, such as data preparation and feature engineering workflows, but the paper does not demonstrate that transfer.
  • Because the chatbot mixes provenance with the LLM's background knowledge, there is a risk of drift from the record; a human-factors study would reveal whether users notice such drift and whether it matters.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes an interactive explainer agent that co-constructs explanations of an AI system's outputs with the user, using data provenance as the grounding source. It describes a prototype ENEXA chatbot for neural class expression learning, in which provenance metadata is added to prompts and an LLM generates responses. It also introduces an evaluation framework based on LLM-simulated user personas and an LLM-as-a-judge, and reports an experiment with six personas, ten class expressions, and three-turn dialogues, scored on seven criteria. The authors position the work as a vision statement with an initial prototype and a scalable evaluation framework, not as a finished deployment.

Significance. If the framework were validated, this work would be a useful contribution to trace-based XAI by operationalizing co-construction of explanations and by proposing a scalable, reproducible evaluation method that combines user simulation with LLM judging. The paper makes a good-faith effort to connect to existing evaluation taxonomies and to acknowledge limitations; it also publicly links the source code and prompts, which supports reproducibility. The prototype illustrates a plausible mechanism for tailoring provenance-based explanations to different user roles. However, the current evaluation does not establish the central two-part claim, because provenance grounding is never directly measured and the LLM-based evaluation instruments are not calibrated against human judgements.

major comments (3)
  1. [Section 4 and Table 1] None of the seven criteria in Section 4 (Clarity & Structure, Depth & Completeness, Correctness & Fidelity, Relevance & Focus, Appropriateness for the Persona, Transparency, Engagement & Intuition) checks whether the assistant's statements are consistent with the provenance metadata. The definition of 'Correctness & Fidelity' is faithfulness to 'the original query/context,' not to the provenance data, and the judge is not described as receiving the metadata as ground truth. Consequently, the high 'Correctness & Fidelity' scores in Table 1 support at most dialogue fluency and topical relevance, not the 'grounded in data provenance' half of the abstract's central claim. The evaluation protocol should either add an explicit provenance-grounding criterion, have the judge verify statements against a provenance store, or both.
  2. [Sections 4 and 5] The evaluation framework's validity depends on two unvalidated proxies: LLM-simulated personas and an LLM-as-a-judge. Section 4 cites prior work showing agreement between LLM judges and humans and cites user-simulation methods, but the paper does not calibrate its own judge or personas against human judgements, and it does not include a baseline or comparison system. The experiment itself is small (six personas, ten class expressions, three turns) and Table 1 reports only means and standard deviations without any uncertainty or significance assessment. The paper's own caveats in Section 5 ('not necessarily substitute for true human behaviour') and Section 6 (deferring correlation with application-grounded metrics to future work) are appropriate, but they mean the results should be described as illustrative, not as evidence that co-construction works.
  3. [Section 6] The claim that the framework enables 'large-scale reproducible evaluations' is stronger than the evidence supports. Reproducibility is aided by the public code, but the evaluation criteria lack a rubric tying each 5-point score to observable dialogue properties, and the framework's validity for ranking different explanation approaches is not established. Adding a concrete scoring rubric, a small human-judgement validation study, and at least one baseline would make the framework's claims testable.
minor comments (4)
  1. [Section 1] The word 'interatively' in the third sentence of the introduction appears to be a typo for 'interactively.'
  2. [Table 1] The table header contains 'Relevances,' which should be 'Relevance' to match the other criterion names.
  3. [Section 4] The seven criteria are listed with one-sentence definitions, but no rubric is provided for assigning values on the 5-point scale; specifying what distinguishes a score of 3 from a 4 on each criterion would improve reproducibility.
  4. [Figure 3] In the example trace, the assistant states that information 'was verified from Wikipedia,' but the provenance metadata mentioned in Section 3 concerns the information extraction pipeline; the example should make clear which provenance records support this claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's evaluation gap is a construct-validity issue, not a circular reduction.

full rationale

The paper makes no formal derivation; it presents a prototype and an evaluation framework. The prototype injects provenance metadata into LLM prompts (Section 3), and the evaluation framework adopts LLM-as-a-judge and user simulation from external prior work [28, 4], explicitly flagging validity concerns [26] and noting that personas do not substitute for true human behaviour (Section 5). There is no fitted parameter renamed as a prediction, no uniqueness theorem imported from the authors, and no ansatz smuggled in via self-citation. The closest self-citations ([6], [20]) are not load-bearing. The genuine weakness is that the 'Correctness & Fidelity' criterion (Section 4) asks whether the explanation is 'faithful to the original query/context,' not whether it matches the provenance metadata, so Table 1 cannot directly support the 'grounded in data provenance' conjunct. That is a validation gap rather than circularity: the scores are not defined in terms of the claims they support, nor are the claims derived from the scores by construction. Therefore the paper should be scored 0 on circularity.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper introduces no fitted parameters or invented entities. The empirical claims rest on three domain assumptions: LLM judgment quality, persona fidelity, and provenance as correctness ground truth. Experimental settings (three turns, six personas, seven criteria) are hand-chosen and should be treated as configurable parameters rather than inferred values.

free parameters (3)
  • Maximum interaction turns = 3
    The evaluation limits user-agent dialogue to three turns (Section 5), which likely affects depth and engagement scores.
  • User persona set = 6 personas
    Six hand-crafted personas are used as simulated users; no sampling rationale or coverage justification is given (Section 5).
  • Evaluation criteria = 7 criteria, 5-point scale
    The seven judgement criteria are authored by the researchers without validation against human-derived constructs (Section 4).
assumptions (3)
  • domain assumption LLM-as-a-judge is a valid proxy for human evaluation
    Section 4 adopts the LLM-as-a-judge approach from [28] and notes concerns but does not calibrate the judge on this task; the correctness and fidelity scores in Section 5 rely on it.
  • domain assumption LLM-simulated personas represent target user groups
    Section 4 builds on user simulation literature; Section 5 concedes personas are not substitutes for true human behaviour, yet the experiments use them as the only user signal.
  • domain assumption Provenance metadata is the appropriate grounding for explanation correctness
    The prototype feeds provenance into prompts, but the judge never verifies statements directly against the provenance store; the fidelity criterion assumes alignment with metadata is the right standard.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Co-constructing Explanations for AI Systems using Provenance." pith.science (2026). https://pith.science/paper/YOUJTJ3O

@misc{pith2026250717761,
  author       = {Pith},
  title        = {Pith review of: Co-constructing Explanations for AI Systems using Provenance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YOUJTJ3O}},
  note         = {Machine review of arXiv:2507.17761}
}
read the original abstract

Modern AI systems are complex workflows containing multiple components and data sources. Data provenance provides the ability to interrogate and potentially explain the outputs of these systems. However, provenance is often too detailed and not contextualized for the user trying to understand the AI system. In this work, we present our vision for an interactive agent that works together with the user to co-construct an explanation that is simultaneously useful to the user as well as grounded in data provenance. To illustrate this vision, we present: 1) an initial prototype of such an agent; and 2) a scalable evaluation framework based on user simulations and a large language model as a judge approach.

Figures

Figures reproduced from arXiv: 2507.17761 by the authors.

Figure 2
Figure 2. Sequence diagram for the ENEXA chatbot used for [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Example of a chat evaluated with the persona ‘data [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

29 extracted references · 17 canonical work pages

  1. [22]

    Katharina J Rohlfing et al. 2020. Explanation as a social practice: toward a conceptual framework for the social design of ai systems. IEEE Transactions on Cognitive and Developmental Systems , 13, 3, 717–728

    work page 2020

  2. [14]

    Amruta Kale, Tin Nguyen, Frederick C Harris Jr, Chenhao Li, Jiyin Zhang, and Xiaogang Ma. 2023. Provenance documentation to enable explainable and trustworthy ai: a literature review. Data Intelligence, 5, 1, 139–162

    work page 2023

  3. [1]

    Ariful Islam Anik and Andrea Bunt. 2021. Data-centric explanations: explain- ing training data of machine learning systems to promote transparency. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI ’21) Article 75. Association for Computing Machinery, Yokohama, Japan, 13 pages. isbn: 9781450380966. doi: 10.1145/3411764.3445736

    arXiv 2021

  4. [2]

    Nagadivya Balasubramaniam, Marjo Kauppinen, Antti Rannisto, Kari Hiekka- nen, and Sari Kujala. 2023. Transparency and explainability of ai systems: from ethical guidelines to requirements. Information and Software Technology, 159, 107197

    work page 2023

  5. [3]

    Alejandro Barredo Arrieta et al. 2020. Explainable artificial intelligence (xai): concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion, 58, 82–115. doi: https://doi.org/10.1016/j.inffus.2019.12.012

    doi:10.1016/j.inffus.2019.12.012 2020

  6. [4]

    Anouck Braggaar, Christine Liebrecht, Emiel van Miltenburg, and Emiel Krah- mer. 2024. Evaluating task-oriented dialogue systems: a systematic review of measures, constructs and their operationalisations. (2024). https://arxiv.org/ab s/2312.13871 arXiv: 2312.13871 [cs.CL]

    work page Pith review arXiv 2024

  7. [5]

    Shruthi Chari, Daniel M Gruen, Oshani Seneviratne, and Deborah L McGuin- ness. 2020. Directions for explainable knowledge-enabled systems. In Knowl- edge Graphs for eXplainable Artificial Intelligence: Foundations, Applications and Challenges. IOS Press, 245–261

    work page 2020

  8. [6]

    Enrico Daga and Paul Groth. 2024. Data journeys: explaining ai workflows through abstraction. Semantic Web, 15, 4, (Oct. 2024), 1057–1083. Roberto Confalonieri, Oliver Kutz, Diego Calvanese, Jose M. Alonso, and Shang-Ming Zhou, (Eds.) doi: 10.3233/sw-233407

    work page doi:10.3233/sw-233407 2024

Show all 29 references
  1. [7]

    Caglar Demir and Axel-Cyrille Ngonga Ngomo. 2023. Neuro-symbolic class expression learning. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence (IJCAI ’23) Article 403. Macao, P.R.China, 9 pages. isbn: 978-1-956792-03-4. doi: 10.24963/...

    2023 doi

  2. [8]

    Jan Deriu, Alvaro Rodrigo, Arantxa Otegi, Guillermo Echegoyen, Sophie Rosset, Eneko Agirre, and Mark Cieliebak. 2021. Survey on evaluation methods for dialogue systems. Artificial Intelligence Review, 54, 755–810

    2021

  3. [9]

    Stefan Grafberger, Paul Groth, Julia Stoyanovich, and Sebastian Schelter. 2022. Data distribution debugging in machine learning pipelines. The VLDB Journal, (Jan. 2022). doi: 10.1007/s00778-021-00726-w

    2022 doi

  4. [10]

    Melanie Herschel, Ralf Diestelkämper, and Houssem Ben Lahmar. 2017. A survey on provenance: what for? what form? what from? The VLDB Journal, 26, 881–906

    2017

  5. [11]

    Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daumé, Miro Dudik, and Hanna Wallach. 2019. Improving fairness in machine learning systems: what do industry practitioners need? In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19). Associati...

    2019

  6. [12]

    John J Horton. 2023. Large language models as simulated economic agents: What can we learn from homo silicus? Tech. rep. National Bureau of Economic Research

    2023

  7. [13]

    Tiancheng Hu and Nigel Collier. 2024. Quantifying the persona effect in LLM simulations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Lun-Wei Ku, Andre Martins, and Vivek Srikumar, (Eds.) Association for Co...

    2024 doi

  8. [15]

    Luca Lauro, Pasquale Leonardo Lazzaro, Marialaura Lazzaro, Paolo Missier, and Riccardo Torlone. 2024. An llm-guided platform for multi-granular collection and management of data provenance, (Nov. 2024). doi: 10.21203/rs.3.rs-536954 9/v1

    2024 doi

  9. [16]

    Xiao-Hui Li et al. 2020. A survey of data-driven and knowledge-aware explain- able ai. IEEE Transactions on Knowledge and Data Engineering

    2020

  10. [17]

    Raoni Lourenço, Juliana Freire, and Dennis Shasha. 2019. Debugging machine learning pipelines. In Proceedings of the 3rd International Workshop on Data Management for End-to-End Machine Learning (DEEM’19) Article 3. Associ- ation for Computing Machinery, Amsterdam, Netherlands...

    2019

  11. [18]

    Sina Mohseni, Niloofar Zarei, and Eric D. Ragan. 2021. A multidisciplinary survey and framework for design and evaluation of explainable ai systems. ACM Trans. Interact. Intell. Syst. , 11, 3–4, Article 24, (Aug. 2021), 45 pages. doi: 10.1145/3387166

    2021 doi

  12. [19]

    Suhong Moon, Marwa Abdulhai, Minwoo Kang, Joseph Suh, Widyadewi Soedar- madji, Eran Kohen Behar, and David Chan. 2024. Virtual personas for language models via an anthology of backstories. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processin...

    2024 doi

  13. [20]

    2022.Provenance: an introduction to PROV

    Luc Moreau and Paul Groth. 2022.Provenance: an introduction to PROV. Springer Nature

    2022

  14. [21]

    Romila Pradhan, Jiongli Zhu, Boris Glavic, and Babak Salimi. 2022. Interpretable data-based explanations for fairness debugging. In Proceedings of the 2022 International Conference on Management of Data (SIGMOD ’22). Association for Co-constructing Explanations for AI Systems ...

    2022

  15. [23]

    Sebastian Schelter and Julia Stoyanovich. 2020. Taming technical bias in ma- chine learning pipelines. Bulletin of the Technical Committee on Data Engineer- ing, 43, 4

    2020

  16. [24]

    Gesina Schwalbe and Bettina Finzel. 2024. A comprehensive taxonomy for explainable artificial intelligence: a systematic survey of surveys on methods and concepts. Data Mining and Knowledge Discovery , 38, 5, 3043–3101

    2024

  17. [25]

    Arne Seeliger, Matthias Pfaff, and Helmut Krcmar. 2019. Semantic web tech- nologies for explainable machine learning models: a literature review. PRO- FILES/SEMEX@ ISWC, 2465, 1–16

    2019

  18. [26]

    Lindia Tjuatja, Valerie Chen, Tongshuang Wu, Ameet Talwalkwar, and Graham Neubig. 2024. Do LLMs exhibit human-like response biases? a case study in survey design. Transactions of the Association for Computational Linguistics , 12, 1011–1026. doi: 10.1162/tacl_a_00685

    2024 doi

  19. [27]

    Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang

  20. [28]

    Lianmin Zheng et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In Proceedings of the 37th International Conference on Neural Information Processing Systems (NeurIPS ’23) Article 2020. Curran Associates Inc., New Orleans, LA, USA, 29 pages

    2023

  21. [2017]

    In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing

    Men also like shopping: reducing gender bias amplification using corpus- level constraints. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics, Copenhagen, Denmark, (Sept. 2017), 2979–2989. doi:...

    2017 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.