The khipu problem frames a governance failure in distributed AI where interpretive continuity is lost even when traces remain, requiring infrastructure to preserve reading practices rather than only data retention.
super hub Canonical reference
Towards A Rigorous Science of Interpretable Machine Learning
Canonical reference. 71% of citing Pith papers cite this work as background.
102
Pith papers citing it
3,151
external citations · Pith
Background
71% of classified citations
abstract
As machine learning systems become ubiquitous, there has been a surge of interest in interpretable machine learning: systems that provide explanation for their outputs. These explanations are often used to qualitatively assess other criteria such as safety or non-discrimination. However, despite the interest in interpretability, there is very little consensus on what interpretable machine learning is and how it should be measured. In this position paper, we first define interpretability and describe when interpretability is needed (and when it is not). Next, we suggest a taxonomy for rigorous evaluation and expose open questions towards a more rigorous science of interpretable machine learning.
hub tools
citation-role summary
background 13
method 1
citation-polarity summary
representative citing papers
ExplAIner is a layered first-order logic that expresses major XAI explanation queries over Boolean models with evaluation in the Boolean hierarchy and computation in FP^NP.
A systematic survey unifies presentation, digital injection, and GenAI synthesis attacks on identity documents, audits datasets for a reality gap, identifies SDGI in multimodal models, and reports APCER above 25% for top models on synthetic IDs.
Forensic Knowledge Graphs integrate forensic traces, causal dependencies, and scene links via a new authentication network and Iterative Context Refinement to outperform standard detectors and VLMs on detection, localization, and justification.
An argument paper reframes LLM explainability as an embodied, situated practice based on Dourish and enactivist cognition, identifying ontological obstacles in internal explanations and advocating affordance-based designs.
Behavior Forecasters trained on LRM trajectories outperform larger models in predicting repeatability and input sensitivity at low cost.
Concept-based models can use controlled 'benign' information leakage to remain accurate and intervenable under real-world concept incompleteness by reframing their training objective.
Agentic-imodels evolves scikit-learn regressors via an autoresearch loop to jointly boost predictive performance and LLM-simulatability, improving downstream agentic data science tasks by up to 73% on the BLADE benchmark.
ISAAC auditing applied to three DTI models on the Davis benchmark finds 25% relative differences in causal reasoning scores despite nearly identical AUROC values.
In-context symbolic regression methods improve robustness of symbolic formula recovery from KANs, cutting median OFAT test MSE by up to 99.8 percent across hyperparameter sweeps.
Qualitative study of 19 practitioners reveals ten LLM product evaluation practices and introduces the results-actionability gap as a key barrier to turning findings into improvements.
A framework that uses LLM sparse autoencoder features as high-dimensional outcome variables, with new k-FWER controlling bootstrap tests and formalized autointerp evaluation, to make interpretable discoveries from unstructured data.
A training-free method using Fourier-parameterized star-convex contours optimized via gradients to generate compact, faithful visual attributions for image classifiers on benchmarks like ImageNet.
A method automatically constructs a causal model from behavior tree structure and domain knowledge to generate real-time causal counterfactual explanations for robot decisions.
SAE-NOs extend sparse autoencoders to function spaces via Fourier neural operators with concept and domain sparsity, learning localized patterns more efficiently and generalizing across discretizations on vision data.
MIMIC is a new inversion framework that recovers visual concepts from VLM internal states using joint inversion, feature alignment, and three regularizers.
Chain-of-thought explanations in LLMs are frequently unfaithful: models systematically omit mention of biasing prompt features that change their answers and instead produce rationalizations for those biased outputs.
Activation steering of SAE-attributed components lets practitioners move from correlational inspection to causal hypothesis testing on CLIP failures, with trust shifting to observed model responses (N=8 experts).
AI-native games require runtime generative AI as a non-substitutable core-loop mechanism; a 53-game corpus clusters in language-forward narrative and epistemic designs.
Matryoshka Sparse Autoencoders applied to matrix-factorization embeddings from the Amazon Fashion dataset recover hierarchical monosemantic features that align with metadata and permit targeted intervention.
Introduces a constraint-satisfaction algorithm and complexity results for recovering linear utilities and latent group bonuses to explain observed rankings under hidden sensitive features.
I-SAFE is a post-hoc auditing framework that applies quantile-based and Wasserstein coherence metrics to evaluate distributional response of DTI prediction models under structural perturbations from external priors like KLIFS annotations.
AI models misalign with humans on concept boundaries when probed with implausible category members, such as classifying words as vehicles or vegetables as fruit.
p-ResNet-50 adds a prototype layer with anchor- and medoid-based regularizations to ResNet-50, achieving ROC-AUC 0.994 and accuracy 0.957 on ~12k XCT patches while supplying case-based explanations aligned to expert categories.
citing papers explorer
-
Evolutionary Rule Extraction from Corporate Default Prediction Models
ML classifiers outperform logistic regression on 50k Italian SME defaults 2015-2024; DEXiRE-EVO extracts rules on liquidity erosion, leverage, inefficiency, and macro persistence.
-
Market Regime Council for Dynamic Credit Assignment in Multi-Agent LLM Decision Systems
MRC computes coalition Shapley credits from performance histories to weight three LLM agents, stabilized by Bayesian mixture and regime multipliers, achieving SR 1.51 and 440.1% cumulative return over 1037 days on 13 crypto assets.
-
ChainzRule: Sample-Efficient, Robust Deep Learning Across Tabular, NLP, and Vision Tasks
ChainzRule uses learnable polynomial layers with differential regularization on the Jacobian to promote stable low-frequency representations, claiming improved sample efficiency and robustness on multiple benchmarks.
-
SINAPSE: A lightweight deep learning framework for accurate and explainable neutron-$\gamma$ discrimination
SINAPSE uses a dual-branch neural network with a 1D convolutional autoencoder for denoising and a classifier for neutron-gamma discrimination, trained via random augmentations on high-SNR data and validated with SHAP explanations.
-
Evaluating the False Trust Engendered by LLM Explanations
LLM reasoning traces and post-hoc explanations increase false trust in incorrect predictions, whereas contrastive dual explanations enhance users' ability to distinguish correct from incorrect AI outputs.
-
NeuroViz: Real-time Interactive Visualization of Forward and Backward Passes in Neural Network Training
NeuroViz offers interactive real-time visualization of neural network forward and backward passes, achieving top usability scores in a study with 31 participants compared to existing tools.
-
CoAX: Cognitive-Oriented Attribution eXplanation User Model of Human Understanding of AI Explanations
Cognitive models of user reasoning strategies with XAI methods on tabular data fit human forward-simulation decisions better than ML baselines and support hypothesis testing without new user studies.
-
X-NegoBox: An Explainable Privacy-Budget Negotiation Framework for Secure Peer-to-Peer Energy Data Exchange
X-NegoBox is a proposed explainable framework that negotiates privacy budgets for energy data exchange using trust, sensitivity, and purpose factors, with experiments claiming reduced leakage and higher acceptance rates.
-
From Awareness to Intent: Mitigating Silent Driving System Failures through Prospective Situation Awareness Enhancing Interfaces
Prospective situation awareness enhancing interfaces delivered via AR HUD improve takeover performance after silent automation failures, with perceptual cues most effective at raising situational awareness and system-intent messages best at building trust.
-
Domain-Specialized Object Detection via Model-Level Mixtures of Experts
Model-level MoE of domain-specialized YOLO detectors with gating network outperforms standard ensembles on BDD100K while revealing expert specialization.
-
Interpretable and Explainable Surrogate Modeling for Simulations: A State-of-the-Art Survey and Perspectives on Explainable AI for Decision-Making
This survey synthesizes XAI methods with surrogate modeling workflows for simulations and outlines a research agenda to embed explainability into simulation-driven design and decision-making.
-
Governed Reasoning for Institutional AI
Cognitive Core uses nine typed cognitive primitives, a four-tier governance model with human review as an execution condition, and an endogenous audit ledger to reach 91% accuracy with zero silent errors on prior authorization appeals, outperforming ReAct and Plan-and-Solve baselines.
-
Explainability and Certification of AI-Generated Educational Assessments
A framework using self-rationalization, attribution analysis, and a certification metadata schema with traffic-light workflow enables transparent, audit-ready AI-generated educational assessments aligned to Bloom's and SOLO taxonomies.
-
Agentic AI for Cybersecurity: A Meta-Cognitive Architecture for Governable Autonomy
A meta-cognitive agentic framework coordinates specialized cybersecurity agents through a judgment mechanism to improve decision quality under uncertainty and noise on standard benchmarks.
-
A Neuro-Symbolic Framework for Accountability in Public-Sector AI
A framework combining legal ontology, rule extraction, and solver reasoning verifies whether AI explanations for CalFresh eligibility align with statutory constraints.
-
Why Johnny Can't Use Agents: Industry Aspirations vs. User Realities with AI Agents
Industry markets AI agents for orchestration, creation, and insight, but a usability study with 31 participants reveals users face challenges from capability misalignment and lack of meta-cognition in tools like Operator and Manus.
-
Decoding the Multimodal Maze: A Systematic Review on the Adoption of Explainability in Multimodal Attention-based Models
A systematic review of 55 papers finds explainability for multimodal attention-based models is dominated by attention-weight visualizations, while evaluation remains mostly qualitative and non-standardized.
-
Detection of Real-world Driving-induced Affective State Using Physiological Signals and Multi-view Multi-task Machine Learning
A multi-view multi-task ML method detects real-world driving-induced affective states using physiological signals by modeling inter-drive variability, with results showing performance gains on three datasets.
-
Optimal Explanations of Linear Models
An optimization framework decomposes linear models into increasing-complexity sequences using coordinate updates to generate parametrized interpretability metrics.
-
A Human-Grounded Evaluation of SHAP for Alert Processing
Human-grounded evaluation finds no significant performance improvement from adding SHAP explanations to model confidence scores in alert processing.
-
Do Transformer Attention Heads Provide Transparency in Abstractive Summarization?
Analysis of transformer attention heads in abstractive summarization shows specialization in some heads and proposes a method to measure model reliance on learned attention distributions.
-
Interpretable Question Answering on Knowledge Bases and Text
Compares LIME, input perturbation and attention for explaining QA on KB+text; proposes automatic evaluation paradigm and finds input perturbation superior in both automatic and human studies.
-
CW-B: Class Weighted Boosting Framework for Imbalance Resilient Multi Class Cardiac Phenotyping
CW-B is a class-weighted XGBoost method with missingness indicators and classwise auditing that reports best-in-baseline Accuracy, Macro-F1, Balanced Accuracy, and Prioritized F1 on five-class cardiac phenotyping under imbalance.
-
Cognitive Digital Twins: Ethical Risks and Governance for AI Systems That Model the Mind
Defines cognitive digital twins and introduces a 5A governance framework to address risks such as misrepresentation and proxy action in AI systems that model individual cognition.
-
Analysing drivers and interdependencies in European electricity markets using XAI
DNNs plus SHAP/SSHAP applied to 39 European bidding zones identify solar and gas as key price drivers and simulate a single-price EU market.
-
Beyond Prediction: Longitudinal Reasoning in EHR-Integrated Clinical AI
A coding framework analysis of EHR-integrated clinical AI systems shows predominant use of encounter-level data with limited longitudinal reasoning features such as trajectory modeling and absence reasoning.
-
Beyond Post-hoc Explanation: Toward Glassbox AI via Probabilistic Mediation
The paper proposes the Glassbox Framework in which Bayesian networks serve as transparent ante-hoc mediation layers for generative models to enable auditable reasoning traces and contestable outputs.
-
Artificial Adaptive Intelligence: The Missing Stage Between Narrow and General Intelligence
Proposes Artificial Adaptive Intelligence as the regime between narrow and general AI, defined by elimination of human-specified hyperparameters, and introduces an adaptivity index plus parametric minimality principle grounded in minimum description length.
-
Explanation-Aware Learning for Enhanced Interpretability in Biomedical Imaging
Adding explanation supervision to training improves spatial alignment of saliency maps with clinical annotations on chest X-rays while keeping predictive accuracy comparable.
-
LLMs Should Not Yet Be Credited with Decision Explanation
LLMs support decision prediction and rationale generation but lack evidence for genuine decision explanation, requiring stricter standards to avoid over-crediting.
-
From Trust to Appropriate Reliance: Measurement Constructs in Human-AI Decision-Making
A literature review shows that constructs for appropriate reliance on AI are fragmented, presents three views on the topic, and calls for consensus on objective metrics to enable better comparisons across studies.
-
Out of Context: Reliability in Multimodal Anomaly Detection Requires Contextual Inference
Multimodal anomaly detection must be reframed as cross-modal contextual inference that separates context from observations to define abnormality conditionally.
-
Explainable Human Activity Recognition: A Unified Review of Concepts and Mechanisms
The paper delivers a mechanism-centric taxonomy and unified perspective on explainable human activity recognition methods across sensing modalities.
-
Event-Centric World Modeling with Memory-Augmented Retrieval for Embodied Decision-Making
ERA encodes environments as semantic events, retrieves prior event–action pairs from a knowledge bank, and forms actions by weighted aggregation for interpretable embodied control.
-
DenoGrad: A Gradient-Based Framework for Data Refinement in Tabular and Time-Series Learning
DenoGrad refines noisy tabular and time-series data by optimizing inputs via gradients from a fixed model, yielding better downstream predictions on ten real-world datasets while preserving data statistics.
-
Use of What-if Scenarios to Help Explain Artificial Intelligence Models for Neonatal Health
AIMEN trains an ensemble of neural networks on CTGAN-augmented data to predict adverse labor outcomes at 0.784 F1 and produces sparse counterfactual explanations identifying changes in two to three attributes.
-
Evaluating Physician-AI Interaction for Cancer Management: Paving the Path towards Precision Oncology
A within-subjects study with 32 physicians using a web-based CDSS for 12 synthetic multiple myeloma scenarios found over-reliance on ML outputs when discordant with RCT evidence and poor retention of model validation details.
-
Toward Human-Centered Multi-Agent Systems: Integrating Cognition, Culture, Values, and Cooperation in AI Agents
A survey arguing that multi-agent AI systems remain task-centric and lack integrated computational models of human cognition, culture, values, and social cooperation.
-
Exploration of Perceptual Speech Features for Clinical Decision-Support in Mental Health Care
A systematic analysis of perceptual speech features finds stable associations with symptom severity in depression, anxiety, and ADHD across multiple datasets using XGBoost with SHAP and LIME.
-
A Blueprint for AI-Driven Software Quality: Integrating LLMs with Established Standards
Survey mapping LLM applications in software quality assurance to established standards including ISO/IEC 12207, ISO 25010, CMMI, and TMM, with case studies, challenges, and future directions.
-
The Role of Cooperation in Responsible AI Development
Competitive pressures in AI development create collective action problems that may require industry cooperation, with key factors and strategies identified to enable responsible outcomes.
-
The Mass, Fake News, and Cognition Security
The paper defines Cognition Security (CogSec) as a multidisciplinary field studying cognitive impacts of fake news and outlines research challenges, techniques, and future directions.
-
Unexplainability and Incomprehensibility of Artificial Intelligence
Advanced AI systems are unexplainable in full and produce explanations that humans cannot comprehend.
-
Platonic Projection Structures: Operator-Induced Observability in Representation Learning
The paper introduces 'Platonic Projection Structures,' a reformulation of standard PSD operator theory applied to representation learning, with experiments that verify definitions rather than test predictions.
-
AI Researchers Must Help Lead Arms Control to Mitigate Military AI Risks
AI researchers must lead technical research in arms control to mitigate risks from military AI systems, drawing lessons from nuclear deterrence.
-
LLM Agent Based Renewable Energy Forecasting Using Edge and IoT Data A Review of Solar Wind Weather and Grid Aware Decision Support
A survey of LLM agent applications in renewable energy forecasting proposing a six-layer taxonomy and listing twelve open challenges.
-
The frame problem in quantitative practice: ontological uncertainty and epistemic humility in an age of automated inference
A synthetic review arguing that frame (ontological) uncertainty is structurally invisible within quantitative models and drives most consequential failures in automated inference.
-
On the Semantic Interpretability of Artificial Intelligence Models
This survey classifies semantic interpretability methods in AI models by nature and feature introduction, reviews user impact, and identifies remaining gaps.
-
Efficacy Analysis in Clinical Trials: A Comprehensive Review of Statistical and Machine Learning Approaches
A review summarizing parametric, nonparametric, Bayesian, and machine learning methods for efficacy analysis in clinical trials and identifying gaps such as high-dimensional data and missingness.
- Signed Evidence Flow: Conflict-Aware and Stability-Calibrated Data Analysis