Pith. sign in

REVIEW 3 major objections 5 minor 26 references

Shapley values meet audio: a framework for explaining multimodal AI

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-05 08:30 UTC pith:BJ46YYMH

load-bearing objection Open-source toolkit for Shapley-value explainability of text+audio MLLMs; empirical claims are preliminary and partly confounded by SGPA segmentation. the 3 major comments →

arxiv 2606.07533 v1 pith:BJ46YYMH submitted 2026-04-21 cs.CL cs.AIcs.SD

Bridging Traditional Explainability Methods and Multimodal Multilingual Models: An XAI-Based Analysis

Pawe{\l} Pozorski , Jakub Muszy\'nski , Maria Ganzha This is my paper
classification cs.CL cs.AIcs.SD
keywords multimodalaudiocomputationalframeworkcomputationexplainabilityextensioninteractive
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper extends Shapley value (SV) attribution — a cooperative-game-theoretic method for assigning credit to input features — to multimodal large language models that process both text and audio. The central obstacle is that audio, unlike text, lacks natural semantic boundaries: a few seconds of speech can produce hundreds of dense encoder frames, making the coalition space (which grows as 2^n) computationally intractable. To resolve this, the authors introduce Spectrogram-Guided Phonetic Alignment (SGPA), a preprocessing method that uses CTC-based alignment followed by spectrogram-guided boundary refinement to map audio streams onto word-aligned segments, reducing the number of players by 10–50× while preserving interpretability. They pair SGPA with three SV estimation strategies — exact computation for short inputs, Monte Carlo sampling, and Neyman-allocated stratified sampling using complementary contributions — and provide a Python package, experimental infrastructure, and GUI. Experiments on a single audio-capable model (LFM2-Audio-1.5B) across English, Spanish, and French yield two main empirical findings: (1) input modality (text vs. speech) is a primary driver of attribution volatility, with speech inputs producing broader, more variable attribution distributions than text; and (2) standard syntactic importance proxies (dependency fan-out, root alignment) poorly predict which tokens receive high Shapley values, and this misalignment varies by language. The authors explicitly acknowledge that SGPA is not attribution-neutral — it redefines the cooperative game by changing the player partition from model-native audio units to externally aligned word segments — but argue that this trade-off is necessary to make the computation feasible at all.

Core claim

The paper's central technical contribution is SGPA, which makes Shapley value computation tractable for audio by replacing hundreds of native encoder frames with a handful of word-aligned segments. The central empirical finding is that input modality — whether information arrives as text or speech — systematically shifts how the model distributes attribution, with speech producing higher-entropy, more volatile patterns. The paper also demonstrates that linguistic heuristics (syntactic centrality) are unreliable proxies for model attention, and that this unreliability is itself language-dependent.

What carries the argument

Spectrogram-Guided Phonetic Alignment (SGPA): a preprocessing pipeline that (1) decomposes transcripts into characters, (2) uses Wav2Vec2 CTC alignment to estimate character-level timestamps, (3) refines boundaries using short-time energy and spectral flux to avoid cutting through phonetic transitions, and (4) aggregates into word-level segments. This is combined with three SV estimators: exact computation, Monte Carlo with first-order omission, and Complementary Contributions with Neyman allocation. The utility function compares model responses using TF-IDF cosine similarity.

Load-bearing premise

The load-bearing premise is that SGPA's word-aligned segmentation is a meaningful player definition for the cooperative game. The paper itself acknowledges that SGPA 'fundamentally alters attribution statistics by modifying the segmentation of the game.' Since the model internally processes audio using its own tokenization scheme — which need not align with word boundaries — the Shapley values computed over SGPA segments measure a different game than the one defined by the模型-

What would settle it

If SV computed over SGPA word-aligned segments diverges systematically from SV computed over model-native audio units in ways that change qualitative conclusions about modality effects, then SGPA's redefined game is not a faithful proxy for model behavior.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If SGPA is a valid proxy, SV-based explainability can be systematically applied to voice-enabled AI systems, enabling audits of how audio inputs drive model decisions in customer service, healthcare, and other deployment scenarios.
  • The finding that modality drives attribution volatility suggests that text-only explainability studies may not transfer to speech-input settings — models may process the 'same' information differently depending on whether it arrives as text or audio.
  • The language-specific misalignment between syntactic importance and attribution scores implies that explainability metrics developed for English may not generalize, requiring language-specific interpretability frameworks.
  • The interleaving asymmetry finding — that early modality acts as an attribution anchor in mixed text-audio inputs — has direct implications for prompt design in multimodal systems.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This manuscript presents a framework for computing Shapley value (SV) attributions for multimodal large language models (MLLMs) that process both text and audio. The work introduces three main contributions: (1) the mllm-shap Python package implementing multiple SV approximation strategies (exact, Monte Carlo, complementary contributions, and Neyman allocation) adapted for variable-length multimodal sequences; (2) Spectrogram-Guided Phonetic Alignment (SGPA), a preprocessing method that reduces audio coalition space by mapping dense audio tokenizations to word-aligned segments; and (3) empirical experiments on the LFM2-Audio-1.5B model across modality configurations (text/speech input-output) and languages (English, Spanish, French), finding that input modality drives attribution volatility and that syntactic importance proxies poorly predict model attention. The paper also provides a companion GUI and reproducible experimental infrastructure with checkpointing and artifact versioning.

Significance. The manuscript makes a genuine engineering contribution by providing an open-source, pip-installable package (mllm-shap) with a GUI for multimodal SV computation, along with published datasets and reproducible infrastructure. The SGPA method addresses a real computational bottleneck (10-50x coalition reduction), and the authors deserve credit for transparently documenting SGPA's non-neutrality in Section 6.6 with controlled diagnostics (Table 6.4). The Neyman estimator benchmarking (Section 6.5) against exact SV ground truth on 27 samples provides falsifiable convergence evidence. However, the empirical claims rest on a single model with limited sample sizes, and the central comparative claim about modality effects is confounded by an asymmetric segmentation design that the paper acknowledges but does not fully resolve.

major comments (3)
  1. §9.2.2, Table 9.5: The headline claim that 'input modality is a primary driver of attribution volatility' rests on comparing T2T (native subword tokenization, ~7-27 tokens/sample per Table 8.2) against S2T/S2S (SGPA word-aligned segments, ~5-10 segments/sample). These are structurally different cooperative games with different player counts and granularities. The paper's own §6.6 diagnostics (Table 6.4) confirm SGPA significantly alters attribution statistics (normalized entropy p<0.01, Cohen's d=-1.37 for SM2S). Since SGPA is applied only to audio, the observed entropy shift between T2T and speech modes conflates genuine modality effects with segmentation granularity differences. The one metric showing significant cross-modality differences (normalized entropy, Table 9.5) is precisely the metric most sensitive to partition structure, while Gini and top-20-mass—which are more length-rob—
  2. §7, normalized entropy definition: The paper normalizes entropy by √n to address length differences, but this does not correct for the structural difference in game partition between subword tokens and word-level segments. A concrete control is needed: a condition where text is also segmented at word level (matching SGPA granularity) would allow disentangling modality effects from segmentation artifacts. Without this control, the modality claim cannot be separated from the segmentation confound. The authors should either add this control or substantially soften the claim from 'input modality is a primary driver' to 'input modality and segmentation granularity jointly affect attribution patterns.'
  3. §9.2-9.3, Tables 9.5 and 9.6: Many statistical tests yield non-significant results or NaN entries due to normality violations (Table 9.5: 4 of 12 Gini/top-20-mass entries are NaN; all multilingual comparisons in §9.3.2 yield non-significant p-values). The paper acknowledges these limitations in §9.2.3 and §9.3.3, but the Summary (§10) still states that 'input modality is a significant driver of attribution volatility' without adequate qualification. The claims in §10 should be brought into closer alignment with the statistical evidence actually presented in Chapter 9.
minor comments (5)
  1. §5.2.1, Eq. (5.6): The SGPA boundary cost weights α=0.8 and β=0.2 are described as 'empirically determined' on English samples, with a note that cross-lingual investigation is needed. Since experiments include Spanish and French, some discussion of whether these weights transfer would strengthen the paper.
  2. Table 9.2: The linguistic analysis assigns 'apt' and 'cha' (subword fragments of 'Recaptcha') the dependency label ROOT and dobj respectively, which appears to be a spaCy parsing artifact from subword tokenization. This should be noted as a limitation of applying dependency parsing to subword units.
  3. §5.1.1: The utility function U3 uses TF-IDF over the set of all model responses D={R_S | S⊂N}, which means the vocabulary and IDF weights change as the sampling budget changes. This dependency should be noted as a potential source of instability in approximate SV.
  4. The manuscript appears to be a thesis (with 'student record book numbers' and a thesis supervisor), which explains its length and structure. The AI usage declaration is commendable in its transparency.
  5. Several references have incomplete formatting (e.g., 'Foundation 2025' for the TF-IDF Wikipedia citation).

Simulated Author's Rebuttal

3 responses · 1 unresolved

We thank the referee for a careful and substantive review. The referee raises three major comments, all concerning the same core issue: the confounding of modality effects with segmentation granularity differences (SGPA word-aligned segments for audio vs. native subword tokenization for text) in our headline claim about input modality driving attribution volatility. We agree with the substance of all three comments and will revise the manuscript accordingly. Specifically, we will (1) soften the modality claim in the Summary to acknowledge that segmentation granularity and modality are jointly confounded, (2) add discussion of the limitation that a word-level text control condition is needed to fully disentangle these effects, and (3) align the Summary claims with the statistical evidence actually presented. We cannot run the proposed word-level text control experiment within the current revision cycle due to compute constraints, which we acknowledge as a standing limitation.

read point-by-point responses

  1. Referee: §9.2.2, Table 9.5: The headline claim that 'input modality is a primary driver of attribution volatility' rests on comparing T2T (native subword tokenization, ~7-27 tokens/sample) against S2T/S2S (SGPA word-aligned segments, ~5-10 segments/sample). These are structurally different cooperative games with different player counts and granularities. The paper's own §6.6 diagnostics confirm SGPA significantly alters attribution statistics. Since SGPA is applied only to audio, the observed entropy shift between T2T and speech modes conflates genuine modality effects with segmentation granularity differences. The one metric showing significant cross-modality differences (normalized entropy) is precisely the metric most sensitive to partition structure, while Gini and top-20-mass are more length-robust.

    Authors: The referee is correct on all points. The comparison between T2T (native subword tokenization) and speech modes (SGPA word-aligned segments) does conflate modality effects with segmentation granularity differences, and our own §6.6 diagnostics (Table 6.4) confirm that SGPA significantly alters attribution statistics including normalized entropy (Cohen's d = -1.37 for SM2S). We acknowledge that normalized entropy is indeed the metric most sensitive to partition structure, which is precisely the metric that shows significant cross-modality differences in Table 9.5. The more length-robust metrics (Gini, top-20-mass) either violate normality assumptions or yield non-significant results. This is a genuine confound that we cannot fully resolve with the current experimental design. We will revise the manuscript to explicitly state this confound in §9.2.2 and §9.2.3, noting that the observed entropy differences may reflect segmentation granularity differences rather than (or in addition to) genuine modality effects. revision: yes

  2. Referee: §7, normalized entropy definition: The paper normalizes entropy by √n to address length differences, but this does not correct for the structural difference in game partition between subword tokens and word-level segments. A concrete control is needed: a condition where text is also segmented at word level (matching SGPA granularity) would allow disentangling modality effects from segmentation artifacts. Without this control, the modality claim cannot be separated from the segmentation confound. The authors should either add this control or substantially soften the claim.

    Authors: We agree that a word-level text segmentation control is the correct experimental design to disentangle modality from segmentation effects, and that √n normalization does not correct for the structural partition difference between subword tokens and word-level segments. We cannot add this control experiment within the current revision cycle due to compute constraints (the experimental infrastructure requires substantial model call budgets, as documented in Appendix F and Table 9.3). However, we will substantially soften the claim throughout the manuscript. Specifically, in the Abstract we will change 'input modality is a primary driver of attribution volatility' to language indicating that input modality and segmentation granularity jointly affect attribution patterns. In §9.2.3 (Conclusions), we will add an explicit discussion of this confound as a primary limitation. In §10 (Summary), we will qualify the modality claim accordingly. We will also add the word-level text control to §11 (Future Work) as a priority experiment. We note that the package infrastructure already supports configurable segmentation, so this experiment is feasible in principle once compute resources are available. revision: partial

  3. Referee: §9.2-9.3, Tables 9.5 and 9.6: Many statistical tests yield non-significant results or NaN entries due to normality violations. The paper acknowledges these limitations in §9.2.3 and §9.3.3, but the Summary (§10) still states that 'input modality is a significant driver of attribution volatility' without adequate qualification. The claims in §10 should be brought into closer alignment with the statistical evidence actually presented in Chapter 9.

    Authors: The referee is correct that the Summary overstates the statistical evidence. In §9.2, only normalized entropy shows significant cross-modality differences (and as discussed above, this metric is confounded by segmentation granularity), while Gini and top-20-mass either violate normality or are non-significant. In §9.3, all multilingual comparisons yield non-significant p-values. The Summary's statement that 'input modality is a significant driver of attribution volatility' does not adequately reflect this evidence base. We will revise §10 to align with the actual statistical findings: we will qualify the modality claim to note that it rests primarily on normalized entropy differences (which are themselves confounded by segmentation), acknowledge that Gini and top-20-mass metrics do not reach significance, and note that multilingual comparisons yielded no significant differences. We will also add explicit qualification that these findings are preliminary and based on a single model with limited sample sizes. revision: yes

standing simulated objections not resolved
  • We cannot run the proposed word-level text segmentation control experiment within the current revision cycle due to compute constraints. The experimental runs require substantial model call budgets (e.g., T2T on multi_sentence averages ~1174 calls per sample, Table 9.3), and our hardware resources (Appendix B) are fully allocated. We acknowledge this as a genuine limitation and will add it to Future Work, but we cannot resolve it now.

Circularity Check

0 steps flagged

No circularity found — derivation chain relies on external theory and standard methods; empirical claims are independently tested

full rationale

The paper's derivation chain is self-contained and non-circular. All load-bearing theoretical components — Shapley value theory (Shapley 1951), MC approximation (Horovicz and Goldshmidt 2024), CC and Neyman allocation (Zhang et al. 2023) — come from external sources with no author overlap. SGPA is a novel preprocessing method defined independently of any experimental claim; it is not a fitted parameter renamed as a prediction. The utility function U3 (TF-IDF cosine similarity) is a standard NLP measure defined independently of the modality conclusions. The headline empirical claim ('input modality is a primary driver of attribution volatility') is supported by paired t-tests on computed SV distributions (Table 9.5), not by definition or by a fitted parameter. The skeptic's concern about SGPA segmentation confounding cross-modality comparisons is a validity/correctness issue — the paper compares different cooperative games (native subword tokens vs. word-aligned segments) — but this is not circularity: SGPA is not defined in terms of the modality claim, and no parameter is fitted to force the conclusion. The paper explicitly discloses this limitation in §6.6 and §4.2. No step in the derivation reduces to its inputs by construction.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 3 invented entities

The free parameters (alpha, beta) are hand-tuned on English samples without cross-lingual validation, which is a limitation for a multilingual study. The core axiom that word-aligned segments are valid SV players is ad hoc to this paper and acknowledged as non-neutral. The utility function choice (TF-IDF cosine) is a domain assumption with known limitations. No invented entities are pulled from thin air; all are grounded in reproducible code or falsifiable benchmarks.

free parameters (5)
  • alpha (SGPA boundary cost weight) = 0.8
    Hand-tuned weight for RMS energy in the boundary refinement cost function (Eq. 5.6). Empirically determined on English samples; cross-lingual impact not investigated.
  • beta (SGPA boundary cost weight) = 0.2
    Hand-tuned weight for spectral flux in the boundary refinement cost function (Eq. 5.6). Same empirical tuning as alpha.
  • delta (SGPA search window) = not stated
    Search window radius for boundary refinement, mentioned but not numerically specified in the method description.
  • m_init (Neyman initial sample count) = max(2, floor(m/(2n^2)))
    Following Zhang et al. 2023; controls the initial sampling phase of the Neyman algorithm.
  • Default sampling budget (3n^2) = 3n^2
    Conservative default budget adopted for Neyman-based explainers based on convergence observations in §6.5.
axioms (4)
  • standard math Shapley value axioms (Efficiency, Symmetry, Null player, Linearity) uniquely define a fair allocation.
    Invoked in §5.1, Table 5.2. Standard cooperative game theory result from Shapley (1951).
  • domain assumption The utility function U(S) = CS(h(R_S), h(R_N)) (TF-IDF cosine similarity of model responses) is a meaningful proxy for model output quality.
    Invoked in §5.1.1. The paper itself cites criticism of cosine similarity (Zhou et al. 2022, Steck et al. 2024) but adopts it for lack of better alternatives.
  • ad hoc to paper Word-aligned audio segments are valid players for the cooperative game, even though they do not correspond to the model's internal audio tokenization.
    Invoked in §5.2 and §6.6. The paper acknowledges this changes the game but argues it is necessary for tractability and interpretability.
  • domain assumption Model responses are sufficiently deterministic under temperature=0, top_k=1 to treat SV variance as dominated by sampling scheme rather than model stochasticity.
    Invoked in §6.5. Standard assumption for SV-based XAI but not formally verified for the LFM2 model.
invented entities (3)
  • Spectrogram-Guided Phonetic Alignment (SGPA) independent evidence
    purpose: Preprocessing method to reduce audio feature count from 150+ to 5-10 word-aligned segments for tractable SV computation.
    SGPA is evaluated against native tokenization in §6.6, showing it changes attribution statistics. The method is falsifiable: one can compare SV computed with and without SGPA and measure the attribution shift, which the paper does.
  • mllm-shap Python package independent evidence
    purpose: Model-agnostic SV computation framework for multimodal LLMs.
    Published on PyPI with open-source code on GitHub. Reproducibility is independently verifiable.
  • Limited Neyman variant independent evidence
    purpose: Combining Neyman allocation over CC coalitions with first-order omission constraint from limited MC.
    Benchmarked against exact SV and standard Neyman in §6.5, showing lower MAE under same budget.

pith-pipeline@v1.1.0-glm · 53335 in / 2761 out tokens · 328664 ms · 2026-07-05T08:30:26.931711+00:00 · methodology

0 comments
read the original abstract

Multimodal Large Language Models (MLLMs) effectively integrate text and audio to interpret context in complex interactive dialogues. However, the internal mechanisms by which heterogeneous modalities influence model behavior remain opaque. While Shapley Values (SV) provide a robust, model-agnostic framework for local explainability in text-based NLP, their extension to multimodal data is hindered by cross-channel dependencies, intricate dialogue structures, and the prohibitive computational complexity of dense audio representations. In this work, we formalize a multimodal extension of the Shapley Value framework, treating discrete text tokens and aligned audio segments as cooperative features. To ensure computational feasibility, we deploy a suite of efficient estimation strategies: exact SV computation for low-dimensional inputs and sampling-based approximations - including Monte Carlo permutations and stratified sampling with Neyman-optimal allocation - to minimize variance under constrained computational budgets. To resolve the granularity mismatch between modalities, we propose Spectrogram-Guided Phonetic Alignment (SGPA), a novel preprocessing method that maps high-frequency audio streams to interpretable, word-aligned segments. Our contribution is twofold: first, we provide an open-source, model-agnostic Python package and a companion GUI for the computation and interactive visualization of multimodal attributions. Second, we evaluate our framework using curated subsets of the VoiceBench and Infinity Instruct datasets across diverse multilingual scenarios. Our experimental results reveal that input modality is a primary driver of attribution volatility and demonstrate that standard syntactic importance proxies often fail to predict model attention in multimodal, cross-lingual contexts.

Figures

Figures reproduced from arXiv: 2606.07533 by Jakub Muszy\'nski, Maria Ganzha, Pawe{\l} Pozorski.

Figure 5.1
Figure 5.1. Figure 5.1: Example decomposition for n = 16 for Large Language Model’s (LLM’s) input tokens. We first group adjacent tokens by 4 (Level 1 - 4 groups in total indicated by different colors), then each of the subgroups is divided into 2 (Level 2 - 8 groups in total, each in different color and frame border). Final Shapley values (SV) are in the lowest frame. All floating-point numbers represent SV calculated for corr… view at source ↗
Figure 5
Figure 5. Figure 5: illustrates the SGPA pipeline, where CTC boundaries are shifted to acoustically [PITH_FULL_IMAGE:figures/full_fig_p035_5.png] view at source ↗
Figure 5.2
Figure 5.2. Figure 5.2: Schematic representation of the Spectrogram-Guided Phonetic Alignment (SGPA) [PITH_FULL_IMAGE:figures/full_fig_p036_5_2.png] view at source ↗
Figure 6
Figure 6. Figure 6: summarizes the execution flow used throughout the experiments. Given a source [PITH_FULL_IMAGE:figures/full_fig_p039_6.png] view at source ↗
Figure 6.1
Figure 6.1. Figure 6.1: High-level package pipeline used in all experiments: chat construction and feature-unit [PITH_FULL_IMAGE:figures/full_fig_p039_6_1.png] view at source ↗
Figure 6.2
Figure 6.2. Figure 6.2: Manual test on correctness of masks generation with token filtering criteria and [PITH_FULL_IMAGE:figures/full_fig_p041_6_2.png] view at source ↗
Figure 6.3
Figure 6.3. Figure 6.3: Beginning of the documentation home page. It includes a brief package description [PITH_FULL_IMAGE:figures/full_fig_p041_6_3.png] view at source ↗
Figure 6.4
Figure 6.4. Figure 6.4: End-to-end experimental infrastructure: configuration expansion, execution engine, [PITH_FULL_IMAGE:figures/full_fig_p042_6_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: ) [PITH_FULL_IMAGE:figures/full_fig_p043_6.png] view at source ↗
Figure 6.5
Figure 6.5. Figure 6.5: Execution Directed Acyclic Graph (DAG) for a single run: build explainer [PITH_FULL_IMAGE:figures/full_fig_p044_6_5.png] view at source ↗
Figure 6.6
Figure 6.6. Figure 6.6: Optional observability: example W&B (Weights&Biases) dashboard panels (progress [PITH_FULL_IMAGE:figures/full_fig_p045_6_6.png] view at source ↗
Figure 6.7
Figure 6.7. Figure 6.7: On-disk artifact layout per run showing spec.json, checkpoint.json, per-sample results, and aggregated metrics. 6.3. Graphical User Interface To bridge the gap between the Shapley value (SV) framework and practical model diagnostics, a companion Graphical User Interface (GUI) was developed. The GUI enables an interactive workflow aligned with the experimental procedures used in Chapter 8: the user config… view at source ↗
Figure 6.8
Figure 6.8. Figure 6.8: High-level system architecture of the interactive Shapley values (SV) explanation tool, [PITH_FULL_IMAGE:figures/full_fig_p047_6_8.png] view at source ↗
Figure 6.9
Figure 6.9. Figure 6.9: Unified Modeling Language (UML) Use Case diagram illustrating the primary [PITH_FULL_IMAGE:figures/full_fig_p048_6_9.png] view at source ↗
Figure 6.10
Figure 6.10. Figure 6.10: Screenshot of the main application interface, demonstrating the layout: (A) Configu [PITH_FULL_IMAGE:figures/full_fig_p049_6_10.png] view at source ↗
Figure 6.11
Figure 6.11. Figure 6.11: Screenshot of the main application interface demonstrating the layout in case of [PITH_FULL_IMAGE:figures/full_fig_p049_6_11.png] view at source ↗
Figure 6
Figure 6. Figure 6: visualizes the aggregated performance metrics across these configurations. This [PITH_FULL_IMAGE:figures/full_fig_p050_6.png] view at source ↗
Figure 6.12
Figure 6.12. Figure 6.12: Validation of package stability across all explainer configuration knobs. The panels [PITH_FULL_IMAGE:figures/full_fig_p050_6_12.png] view at source ↗
Figure 6
Figure 6. Figure 6: summarizes the results for all non-hierarchical estimators. Across sampling [PITH_FULL_IMAGE:figures/full_fig_p051_6.png] view at source ↗
Figure 6.13
Figure 6.13. Figure 6.13: Approximation accuracy of all non-hierarchical methods relative to the exact Shapley [PITH_FULL_IMAGE:figures/full_fig_p052_6_13.png] view at source ↗
Figure 6.14
Figure 6.14. Figure 6.14: Accuracy–runtime trade-off for hierarchical configurations. The horizontal axis [PITH_FULL_IMAGE:figures/full_fig_p052_6_14.png] view at source ↗
Figure 6
Figure 6. Figure 6: presents MAE and relative MAE (RMAE) for both the standard and limited Neyman [PITH_FULL_IMAGE:figures/full_fig_p052_6.png] view at source ↗
Figure 6.15
Figure 6.15. Figure 6.15: Convergence of standard and limited Neyman estimators on a single explanation. [PITH_FULL_IMAGE:figures/full_fig_p053_6_15.png] view at source ↗
Figure 6.16
Figure 6.16. Figure 6.16: Distribution of explainable tokens (%) by Spectrogram-Guided Phonetic Alignment [PITH_FULL_IMAGE:figures/full_fig_p055_6_16.png] view at source ↗
Figure 6.17
Figure 6.17. Figure 6.17: Derivative comparison under Spectrogram-Guided Phonetic Alignment (SGPA) and [PITH_FULL_IMAGE:figures/full_fig_p056_6_17.png] view at source ↗
Figure 6.18
Figure 6.18. Figure 6.18: Positional bias of mean Shapley value (SV) by mode under Spectrogram-Guided [PITH_FULL_IMAGE:figures/full_fig_p057_6_18.png] view at source ↗
Figure 6.19
Figure 6.19. Figure 6.19: Normalized attribution entropy by mode with and without Spectrogram-Guided [PITH_FULL_IMAGE:figures/full_fig_p057_6_19.png] view at source ↗
Figure 9
Figure 9. Figure 9: ) to facilitate the comparison of information accumulation dynamics. Crucially, all [PITH_FULL_IMAGE:figures/full_fig_p060_9.png] view at source ↗
Figure 9.1
Figure 9.1. Figure 9.1: Distribution of explainable tokens per mode. The top chart presents results for [PITH_FULL_IMAGE:figures/full_fig_p070_9_1.png] view at source ↗
Figure 9.2
Figure 9.2. Figure 9.2: Distribution of Shapley values (SV) by mode and importance (Top) or Group [PITH_FULL_IMAGE:figures/full_fig_p072_9_2.png] view at source ↗
Figure 9.3
Figure 9.3. Figure 9.3: E2 diagnostics for VoiceBench__multi_sentence. (Top) Audio share of total attribu￾tion mass (|SV |) by mode; interleaving patterns differ, indicating modality-order asymmetry. (Bottom) Ratio of mean |SV | on text tokens near audio segments (within a small neighborhood) versus far from audio segments; ratios near 1 suggest weak but non-zero local cross-modal coupling. • Entropy Shifts: Speech-integrated i… view at source ↗
Figure 9.4
Figure 9.4. Figure 9.4: The top chart presents the interpolated cumulative sum of Shapley values (SV) by [PITH_FULL_IMAGE:figures/full_fig_p075_9_4.png] view at source ↗
Figure 9
Figure 9. Figure 9: shows the distribution of explainable token counts by language and mode. As in the [PITH_FULL_IMAGE:figures/full_fig_p076_9.png] view at source ↗
Figure 9.5
Figure 9.5. Figure 9.5: Distribution of explainable tokens (%) by language and mode for [PITH_FULL_IMAGE:figures/full_fig_p076_9_5.png] view at source ↗
Figure 9.6
Figure 9.6. Figure 9.6: Correlation between linguistic importance and Shapley values (SV) by mode and [PITH_FULL_IMAGE:figures/full_fig_p077_9_6.png] view at source ↗
Figure 9.7
Figure 9.7. Figure 9.7: Correlation of Shapley values (SV) with linguistic groups by mode and language [PITH_FULL_IMAGE:figures/full_fig_p078_9_7.png] view at source ↗
Figure 9
Figure 9. Figure 9: reports smoothed mean SV over normalized token [PITH_FULL_IMAGE:figures/full_fig_p078_9.png] view at source ↗
Figure 9.8
Figure 9.8. Figure 9.8: Derivative of interpolated cumulative Shapley values (SV) by language and mode for [PITH_FULL_IMAGE:figures/full_fig_p079_9_8.png] view at source ↗
Figure 9.9
Figure 9.9. Figure 9.9: Smoothed Shapley values (SV) over normalized token positions by language and [PITH_FULL_IMAGE:figures/full_fig_p080_9_9.png] view at source ↗
Figure 4
Figure 4. Figure 4: ). Figure 4.3 is an example screenshot from the documentation. Package release notes [PITH_FULL_IMAGE:figures/full_fig_p126_4.png] view at source ↗
Figure 4.1
Figure 4.1. Figure 4.1: Release notes, allowing users to track introduced changes. [PITH_FULL_IMAGE:figures/full_fig_p127_4_1.png] view at source ↗
Figure 4.2
Figure 4.2. Figure 4.2: The documentation page includes illustrative examples with code, explanatory [PITH_FULL_IMAGE:figures/full_fig_p127_4_2.png] view at source ↗
Figure 4.3
Figure 4.3. Figure 4.3: Detailed documentation of part of the BaseMllmChat abstract class, showing the get_conversation and add_audio methods as implemented by the LiquidAudioChat class. It includes input arguments with types, expected output type and description, possible exceptions, and, when available, usage examples or additional comments. 3. Quality tests – These ensure that each package contributor writes high-quality cod… view at source ↗
Figure 5.1
Figure 5.1. Figure 5.1: Unified Modeling Language (UML) Sequence diagram detailing the process of [PITH_FULL_IMAGE:figures/full_fig_p130_5_1.png] view at source ↗
Figure 5.2
Figure 5.2. Figure 5.2: Unified Modeling Language (UML) Sequence diagram detailing the process of [PITH_FULL_IMAGE:figures/full_fig_p131_5_2.png] view at source ↗
Figure 6.1
Figure 6.1. Figure 6.1: Bar plot showing token counts for audio and text in all three datasets used in this [PITH_FULL_IMAGE:figures/full_fig_p134_6_1.png] view at source ↗
Figure 6.2
Figure 6.2. Figure 6.2: Bar plot showing model calls for audio and text in all three datasets used in this [PITH_FULL_IMAGE:figures/full_fig_p134_6_2.png] view at source ↗
Figure 6
Figure 6. Figure 6: and Figure 6.2 [PITH_FULL_IMAGE:figures/full_fig_p135_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

26 extracted references · 26 canonical work pages

  1. [1]

    Tests Figure 4.1: Release notes, allowing users to track introduced changes

    https://pawlo77.github.io/MLLM-Shap/ 128 IV. Tests Figure 4.1: Release notes, allowing users to track introduced changes. Figure 4.2: The documentation page includes illustrative examples with code, explanatory comments, and expected outputs, helping users quickly and clearly understand how to use the API. also made on experiments results, as part of thei...

    work page

  2. [2]

    It validates critical system components, including masking logic, deterministic calculation accuracy, and data integrity

    Automated tests– Integrated into the repository’s Continuous Integration and Con- tinuous Deployment (CI/CD) workflow2 via the pytest3 framework, this suite comprises approximately 400 unit tests. It validates critical system components, including masking logic, deterministic calculation accuracy, and data integrity. Passing the full test suite is a manda...

    work page

  3. [3]

    https://github.com/Pawlo77/MLLM-Shap/actions/runs/20855138582

    work page arXiv

  4. [4]

    https://pypi.org/project/pytest/ 129 D. Package reference Figure 4.3: Detailed documentation of part of theBaseMllmChat abstract class, showing the get_conversation and add_audio methods as implemented by theLiquidAudioChat class. It includes input arguments with types, expected output type and description, possible exceptions, and, when available, usage ...

    work page

  5. [5]

    Quality tests– These ensure that each package contributor writes high-quality code aligned with industry standards such as PEP-8 (Rossum et al. 2001). They utilize tools like flake8, autoflake, isort, and pylint for linting and code quality checks,black for automatic formatting, bandit for security checks, andmypy to ensure type annotations are present th...

    work page 2001

  6. [6]

    https://pypi.org 130 E. GUI reference and requirements alignment This appendix provides the implementation-oriented details of the GUI, including the technol- ogy stack, deployment topology, and a full mapping of features to requirements referenced in Chapter 4. I. Implementation architecture and deployment The application is implemented using a modern cl...

    work page 2013

  7. [7]

    https://www.postgresql.org

    work page

  8. [8]

    GUI reference and requirements alignment Figure 5.1: Unified Modeling Language (UML) Sequence diagram detailing the process of requesting a model load on desired device

    https://nginx.org 131 E. GUI reference and requirements alignment Figure 5.1: Unified Modeling Language (UML) Sequence diagram detailing the process of requesting a model load on desired device. other paths are handled directly as static content. This consolidates access behind a single external port and simplifies future Transport Layer Security (TLS) ha...

    work page

  9. [9]

    https://colab.research.google.com

    work page

  10. [10]

    https://hpc.mini.pw.edu.pl

    work page

  11. [11]

    https://www.plgrid.pl

    work page

  12. [12]

    Computational Resources Analysis II

    https://github.com/Pawlo77/MLLM-Shap/blob/main/paper/pl_grid/wniosek.pdf 135 F. Computational Resources Analysis II. Resource Estimation As the compute requirements represent a significant risk, as outlined in Tables 3.3 and 3.1 in Appendix C, an initial assessment of those requirements was conducted. To estimate the computational resources required for t...

    work page 2024

  13. [13]

    https://pypi.org/project/nltk/

    work page

  14. [14]

    https://pypi.org/project/lingua/

    work page

  15. [15]

    https://huggingface.co/papluca/xlm-roberta-base-language-detection

    work page

  16. [16]

    Dataset construction and preprocessing details segmentation

    https://github.com/WangHelin1997/SpeechTasks 139 G. Dataset construction and preprocessing details segmentation. Finally, Supervised Fine-Tuning (SFT) datasets fulfill the outlined criteria, as they aim to improve model performance in areas where existing knowledge remains incomplete. Utilizing the open-source llm-datasets5 library, theInfinity Instruct(L...

    work page 2025

  17. [17]

    https://github.com/mlabonne/llm-datasets

    work page

  18. [18]

    https://huggingface.co/datasets/Pawlo77/mllm-shap

    work page

  19. [19]

    SS stands for entries included inVoiceBench__single_sentenceand MS -VoiceBench__multi_sentence

    https://huggingface.co/datasets/BAAI/Infinity-Instruct 140 Table 7.1: Summary statistics of the datasets from the original sources. SS stands for entries included inVoiceBench__single_sentenceand MS -VoiceBench__multi_sentence. Note that entries in our final datasets may originate from multiple source datasets simultaneously; therefore, theRowscolumn for ...

    work page

  20. [20]

    Additionally, conversation length was limited to 8 messages, corresponding to 4 human-LLM turns

    The dataset was filtered to include only English, French, and Spanish entries - total644, 478 remain. Additionally, conversation length was limited to 8 messages, corresponding to 4 human-LLM turns. After that we have dataset of637,557rows

    work page

  21. [21]

    Using the predefined topic labels, all entries semantically related toformal logic,logical reasoning,mathematics, andprogrammingwere removed. Smaller models are not expected to handle these topics effectively, therefore that decision has been made in order to reduce bias in that research, as well as to simplify task for TTS pipeline (having it recite math...

    work page

  22. [22]

    Up to 500 characters from concatenated prompts have been input into the system for comparison

    Prompt deduplication was carried out usingSemHash (van Dongen and Tulkens 2025), which employs static embedding models and Approximate Nearest Neighbours (Indyk and Motwani 1998) for efficient semantic text deduplication. Up to 500 characters from concatenated prompts have been input into the system for comparison. Total entries remaining281,737

    work page 2025

  23. [23]

    Any rows with mis- matchedlanguagepredictionscomparedtotheiroriginalannotationswereremoved, resulting in new size of208,270rows

    Non-English entries were verified for correct language classification. Any rows with mis- matchedlanguagepredictionscomparedtotheiroriginalannotationswereremoved, resulting in new size of208,270rows

    work page

  24. [24]

    This yielded our base dataset of62,730observations

    All entries containing sentences of length exceeding 200 characters have been filtered out. This yielded our base dataset of62,730observations. We further create the multi-lingual dataset from this dataset,InfinityInstruct__multi_lingual, which includes single turn conversations from French, Spanish and English (each language has 34 unique entries). Each ...

    work page

  25. [25]

    https://pypi.org/project/sentence-transformers/

    work page

  26. [26]

    All presented entries have similarity exceeding0.7

    https://pypi.org/project/googletrans/ 142 Table 7.3: Partial list of topics related to logic, mathematics, and programming that were filtered out during pre processing. All presented entries have similarity exceeding0.7. logical reasoning deductive reasoning inductive reasoning analogical reasoning causal reasoning common sense reasoning ethical reasoning...

    work page