Pith. sign in

REVIEW 3 major objections 3 minor 13 references

Efficient Quantification of Multimodal Interaction at Sample Level

T0 review · 3 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LSMI gives a per-sample decomposition of multimodal information into redundancy, uniqueness, and synergy using component-wise minima of pointwise surprisals, and claims to be the first estimator that does this efficiently on continuous…

desk verdict Practical sample-level interaction estimator with solid synthetic validation, but the redundancy measure's monotonicity is asserted rather than proven, and negative pointwise values complicate the PID framing. read the letter →

arxiv 2506.17248 v1 pith:KEOVVOUO submitted 2025-06-08 cs.LG cs.AIstat.ML

Zequn Yang , Hongfa Wang , Di Hu This is my paper
classification cs.LGcs.AIstat.ML
keywords multimodalinteractionpartialinformationdecompositionpointwisemutualredundancyuniquenesssynergysample-levelestimationentropy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multimodal data carry three kinds of task-relevant information: redundant information shared by both modalities, unique information available from only one, and synergistic information that appears only when both are used together. This paper's central claim is that all three can be computed for each individual sample, not just averaged over a dataset, and that the computation can be done cheaply on continuous real-world data. The estimator, LSMI, sets sample-level redundancy to the difference between two component-wise minima of pointwise surprisals, and then derives uniqueness and synergy from a pointwise decomposition identity. The authors support the claim with exact matches to ground truth on logic-circuit and Gaussian-mixture benchmarks, with agreement with human judgments on real datasets, and with downstream gains in data partitioning, distillation, and ensembling. If the claim holds, multimodal interaction moves from a dataset-level abstraction to a per-example diagnostic that can guide model training and selection.

What carries the argument

The central object is the pointwise redundancy $r$, built from two component-wise minima of surprisals. Since $i(x;y)=h(x)-h(x|y)$ with $h(x)=-\log p(x)$ and $h(x|y)=-\log p(x|y)$, the paper defines $r^+ = \min(h(x_1), h(x_2))$ and $r^- = \min(h(x_1|y), h(x_2|y))$, then sets $r=r^+-r^-$. The nonnegativity of each surprisal component is what makes each minimum monotone over the event lattice, so the shared component can be quantified as redundancy; the three equations in Eq. 2 then determine $u_1,u_2,s$ from $r$ and the pointwise informations. For continuous data, the entropy values are supplied by a lightweight neural estimator that upper-bounds differential entropy by minimizing a KL divergence, which is why the method scales to real datasets without modeling the joint distribution.

What would settle it

On the noisy-label setup shown in Table 8, check whether any sample satisfies $r+u_1>i(x_1;y)$ or $r+u_2>i(x_2;y)$; either inequality forces a negative uniqueness in Eq. 2 and falsifies the claim that $r$ is a monotone, bounded pointwise redundancy. More directly, any single sample with estimated $r<0$ while both components are nonnegative contradicts the intended redundancy bound.

Watch

Extended reading notes

Core claim

LSMI computes, for every sample $(x_1,x_2,y)$, four numbers $(r,u_1,u_2,s)$ satisfying $i(x_1;y)=r+u_1$, $i(x_2;y)=r+u_2$, and $i(x_1,x_2;y)=r+u_1+u_2+s$. The paper defines redundancy by splitting pointwise mutual information into a positive component $i^+(x;y)=-\log p(x)$ and a negative component $i^-(x;y)=-\log p(x|y)$, taking the minimum of each component across the two modalities, and subtracting: $r = \min(i^+(x_1;y), i^+(x_2;y)) - \min(i^-(x_1;y), i^-(x_2;y))$. Uniqueness and synergy then follow algebraically from the three pointwise mutual informations. The authors claim this is the first estimator that gives reliable sample-level interaction values on continuous real-world data, that it reproduces known ground truth on synthetic systems, and that it is fast because it needs only two per-modality entropy estimators rather than an optimized joint distribution.

Load-bearing premise

The load-bearing premise is that subtracting the two component-wise minima preserves the monotonicity each minimum has on its own, so that the resulting $r$ is a valid pointwise redundancy; the paper offers no proof of this for samples where pointwise mutual information is negative, and negative $r$ values under label noise indicate the premise is not guaranteed.

Editorial extensions

If this is right

  • On synthetic logic circuits, mixtures of Gaussians, and preset-interaction mixtures, LSMI tracks ground-truth redundancy, uniqueness, and synergy more closely than PID-CVX and PID-Batch.
  • Averaged sample-level interactions match human judgments on Food-101 (Pearson 0.98 for redundancy, 0.95 for text uniqueness) and reproduce expected interaction patterns on KS, CMU-MOSEI, and UR-Funny.
  • Fine-tuning an aligned multimodal embedding model on high-redundancy subsets strengthens alignment-driven modalities, while low-redundancy subsets help information-poor modalities, so interaction scores can steer data partitioning.
  • Interaction-weighted distillation and interaction-aware ensembling outperform plain feature distillation and weighted ensembling on KS, UCF, and CREMA-D.
  • LSMI's runtime stays roughly constant as class count grows and is far below PID-Batch (504 s vs 59,680 s on Food-101), because it fits per-modality entropy estimators rather than a joint distribution.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the sign flip of redundancy and uniqueness under label noise shown in Table 8 suggests LSMI could serve as a label-quality probe: samples whose redundancy turns strongly negative are candidates for mislabeling.
  • Beyond the paper, the component-wise minimum construction extends naturally to more than two modalities by taking minima across all sources, so the same estimator could be stress-tested as a higher-order interaction measure, though the authors restrict evaluation to pairwise decompositions.
  • Beyond the paper, a practical robustness check would be to vary the fitted classifiers $p(y|x_1)$, $p(y|x_2)$, and $p(y|x_1,x_2)$ across random seeds and architectures, since the paper reports accuracy but not stability of the interaction values themselves.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper proposes LSMI, a sample-level estimator of multimodal interaction components (redundancy r, uniqueness u1, u2, synergy s) for continuous distributions. The method defines redundancy as the difference of two component-wise minima of pointwise surprisals and conditional surprisals (Eqs. 5-6), derives the remaining components from the linear relations in Eq. (2), and estimates the required entropy terms with the KNIFE differential entropy estimator and learned discriminative models. The authors validate on synthetic circuit-logic, Gaussian-mixture, and preset-interaction datasets, then apply the estimator to several real-world multimodal datasets, including application studies on data partitioning, distillation, and ensembling.

Significance. If the sample-level decomposition were theoretically sound, this would be a useful practical contribution: it is, to my knowledge, the first estimator aimed specifically at sample-level PID-style interaction quantification in continuous settings, it is computationally lighter than distribution-level baselines, and the downstream experiments demonstrate plausible practical value. The paper also ships code and reports runtime comparisons, which are strengths. However, the central theoretical claim that r in Eq. (6) is a valid lattice-compatible redundancy measure is not established, and the current evidence does not directly validate the sample-level quantities that the method advertises.

major comments (3)
  1. [Section 3.2, Eq. (4)] The inference 'both measures, i+ and i-, are compatible with the lattice... Consequently ... r = r+ - r-' is not valid. The set of lattice-compatible functions is closed under minima, but it is not closed under subtraction, and no proof is given that the pointwise difference r inherits the required monotonicity or the inequality in Eq. (3). Table 8 in Appendix A.3.2 reports negative values of r (e.g., -13.719 under label noise), which is consistent with the claimed monotonicity failing at the sample level. Since u1, u2, and s are computed from Eq. (2) once r is fixed, the validity of the entire decomposition rests on this property. The synthetic validation in Section 4.1 compares only averaged R, U, S against distribution-level ground truth, so it cannot detect pointwise violations. Please provide a proof of the lattice properties of r, or a direct per-sample validation against ground-truth pointwise components, or revise the theoretical claims accordingly.
  2. [Section 3.2, Eq. (4)] The paper justifies the decomposition i(x;y) = i+(x;y) - i-(x;y) with i+(x;y) = -log p(x) and i-(x;y) = -log p(x|y) by stating that these components are positive. For continuous distributions, -log p(x) and -log p(x|y) are not guaranteed to be positive; they are negative wherever the density exceeds one. Since LSMI targets continuous data and uses KNIFE differential entropy estimates, the positivity premise is false exactly in the intended regime. The argument that the components are lattice-compatible therefore needs a different basis, or the method needs to be restricted or reinterpreted so that the relevant pointwise quantities are known to be nonnegative.
  3. [Section 4.1.4 and Appendix A.1] The preset-interaction experiments, which are the primary evidence for sample-level accuracy, report only dataset-level averages (R, U, S) in Figure 4. They do not compare per-sample estimates of r, u1, u2, and s with the known per-sample interaction type. Given the paper's central claim is sample-level quantification, this is an important missing validation. Please add a pointwise comparison or report the distribution of sample-level estimates for the synthetic datasets.
minor comments (3)
  1. [Section 3.2, Eq. (3)] The displayed inequality in Eq. (3) appears to state that the shared component is larger than i(x1;y), which is inconsistent with the preceding sentence that redundancy should not exceed the information in any individual source. Please clarify the intended direction or define the notation i((x1;y),(x2;y)).
  2. [Section 3.2] The phrase 'monopoly condition' appears to be a typo for 'monotonicity condition'.
  3. [Algorithm 1 and Eq. (8)] Eq. (8) uses h(y), but the algorithm does not specify how h(y) is computed; please state the estimator used for the target entropy.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: LSMI's sample-level interactions are definitional consequences of an external pointwise PID measure (Ince 2017; Finn & Lizier 2018a) and independently trained entropy estimators, with synthetic validation against external ground truth.

full rationale

The derivation chain is self-contained and none of the load-bearing steps reduces to the paper's own fitted outputs or to self-citation. The redundancy measure in Eq. 6 (r = min(h(x1), h(x2)) - min(h(x1|y), h(x2|y))) is adopted from external prior work (Ince 2017; Finn & Lizier 2018a), not derived by fitting to interaction labels. The entropy estimators h_theta are trained via Eq. 7 only on marginal data p(x1), p(x2) by minimizing D_KL, which is an upper-bound objective independent of the target interactions; Eq. 8 is an algebraic identity for h(x|y). Given r, u1, u2 and s are deterministic solutions of Eq. 2, so their computation is definitional rather than a fitted prediction. Synthetic precision checks are compared against ground truth computed externally via Bertschinger et al.'s PID (Tables 1, Figure 3, Figure 4), so the validation is not circular. The negative per-sample r values in Table 8 indicate a possible failure of the asserted monotonicity/lattice property and a correctness or validity concern, but this is not circularity: the estimator is not fitted to the quantity it claims to predict. Self-citations (e.g., Peng et al. 2022) appear only as related-work context and are not load-bearing for the method's central claim.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new theoretical entities. Its central quantities depend on fitted entropy estimators and discriminative classifiers, which are learned free components, and on the unproven monotonicity of r. The measure itself is borrowed from prior pointwise PID literature.

free parameters (3)
  • entropy estimator parameters theta_1, theta_2 (KNIFE neural network weights) = not reported
    Trained to minimize the KL upper bound in Eq. 7 on data from p(x1) and p(x2); all sample-level estimates of i+ and i- depend on these fitted values.
  • unimodal discriminative models p(y|x1), p(y|x2) = not reported
    Used in Eq. 8 and Algorithm 1 to compute pointwise conditional surprisal i- and hence r, u1, u2, s.
  • multimodal discriminative model p(y|x1,x2) = not reported
    Used to compute i(x1,x2;y) in Algorithm 1 step 6; sample values of synergy depend on it.
assumptions (4)
  • standard math Standard pointwise information identities: Eq. 1 and Eq. 2 hold for the defined PID components.
    The decomposition equations are the conventional PID identities from Williams and Beer and Bertschinger et al., cited in Sections 2 and 3.
  • standard math KNIFE entropy estimators provide a valid upper bound E[h_theta(x)] >= H(X), so minimizing Eq. 7 tightens the bound.
    This follows from the DKL positivity in Eq. 7 and the KNIFE construction, per Pichler et al. (2022).
  • domain assumption Trained discriminative models and entropy estimators approximate the true data distribution closely enough for interaction estimation.
    Section 4.2.2 selects trained models by minimal generalization error, but no guarantee is given that the fitted conditional densities are accurate for the pointwise quantities.
  • ad hoc to paper The redundancy measure r = r+ - r- satisfies the required lattice monotonicity in Eq. 3.
    Asserted in Section 3.2 without proof; negative r values in Table 8 indicate the property is not generally guaranteed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient Quantification of Multimodal Interaction at Sample Level." pith.science (2026). https://pith.science/paper/KEOVVOUO

@misc{pith2026250617248,
  author       = {Pith},
  title        = {Pith review of: Efficient Quantification of Multimodal Interaction at Sample Level},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KEOVVOUO}},
  note         = {Machine review of arXiv:2506.17248}
}
read the original abstract

Interactions between modalities -- redundancy, uniqueness, and synergy -- collectively determine the composition of multimodal information. Understanding these interactions is crucial for analyzing information dynamics in multimodal systems, yet their accurate sample-level quantification presents significant theoretical and computational challenges. To address this, we introduce the Lightweight Sample-wise Multimodal Interaction (LSMI) estimator, rigorously grounded in pointwise information theory. We first develop a redundancy estimation framework, employing an appropriate pointwise information measure to quantify this most decomposable and measurable interaction. Building upon this, we propose a general interaction estimation method that employs efficient entropy estimation, specifically tailored for sample-wise estimation in continuous distributions. Extensive experiments on synthetic and real-world datasets validate LSMI's precision and efficiency. Crucially, our sample-wise approach reveals fine-grained sample- and category-level dynamics within multimodal data, enabling practical applications such as redundancy-informed sample partitioning, targeted knowledge distillation, and interaction-aware model ensembling. The code is available at https://github.com/GeWu-Lab/LSMI_Estimator.

Figures

Figures reproduced from arXiv: 2506.17248 by the authors.

Figure 1
Figure 1. A brief illustration of multimodal interactions [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Event-level redundancy information estimation framework. The proposed measure ensures that information [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Comparison of estimators on data with a mixture [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Comparison of estimators with preset interactions. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Validation on LSMI-based distillation approach. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Comparison between LSMI ensemble and weighted ensemble with various datasets. facilitated by redundant interactions. This alignment pri￾marily benefits information-rich modalities (e.g., Vision in KS, Audio in CREMA-D), enabling more effective learn￾ing while preventin…
Figure 7
Figure 7. Figure 7: Illustration of bivariate Gaussian distributions with different correlation coefficients ( [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Case studies on the Food-101 dataset showing different interaction patterns between visual and textual modalities. [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

13 extracted references · 5 canonical work pages

  1. [4]

    Baseline We adopt three primary types of multimodal learning paradigms: Feature-level fusion: Integration of multiple modalities at the feature level

    A.2. Baseline We adopt three primary types of multimodal learning paradigms: Feature-level fusion: Integration of multiple modalities at the feature level. This includes:Joint learning(Baltru ˇsaitis et al., 2018): A traditional paradigm where features from different modalities are concatenated and jointly mapped into the target space.MMIB (Multimodal Inf...

    work page 2018

  2. [5]

    P., Lyu, Y ., Fan, X., Wu, Z., Cheng, Y ., Wu, J., Chen, L., Wu, P., Lee, M

    Liang, P. P., Lyu, Y ., Fan, X., Wu, Z., Cheng, Y ., Wu, J., Chen, L., Wu, P., Lee, M. A., Zhu, Y ., et al. Multibench: Multiscale benchmarks for multimodal representation learning.arXiv preprint arXiv:2107.07502,

    arXiv

  3. [7]

    R., and Shah, M

    Soomro, K., Zamir, A. R., and Shah, M. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402,

    arXiv

  4. [11]

    T., and Peng, X

    Zhang, Q., Wu, H., Zhang, C., Hu, Q., Fu, H., Zhou, J. T., and Peng, X. Provable dynamic fusion for low-quality multimodal data. InInternational conference on machine learning, pp. 41753–41769. PMLR, 2023a. Zhang, Y ., Latham, P. E., and Saxe, A. A theory of unimodal bias in multimodal learning.arXiv preprint arXiv:2312.00935, 2023b. 12 Efficient Quantifi...

    arXiv 2014

  5. [13]

    The specific model variant employed in our experiments features unimodal branches, each consisting of four Transformer layers

    on KS dataset. The specific model variant employed in our experiments features unimodal branches, each consisting of four Transformer layers. In such architectures, modalities are typically processed through these separate unimodal pathways, with each modality undergoing l layers of dedicated processing before their representations are fused. In our exper...

    work page 2023

  6. [2012]

    H., Liang, P

    Tsai, Y .-H. H., Liang, P. P., Zadeh, A., Morency, L.-P., and Salakhutdinov, R. Learning factorized multimodal representations.arXiv preprint arXiv:1806.06176,

    arXiv

  7. [2014]

    Food-101– mining discriminative components with random forests

    Bossard, L., Guillaumin, M., and Van Gool, L. Food-101– mining discriminative components with random forests. InComputer Vision–ECCV 2014: 13th European Con- ference, Zurich, Switzerland, September 6-12, 2014, Pro- ceedings, Part VI 13, pp. 446–461. Springer,

    work page 2014

  8. [2017]

    Williams, P. L. and Beer, R. D. Nonnegative decom- position of multivariate information.arXiv preprint arXiv:1004.2515,

    arXiv

Show all 13 references
  1. [2018]

    Feature interaction interpretability: A case for explain- ing ad-recommendation systems via neural interaction detection.arXiv preprint arXiv:2006.10966,

    Tsang, M., Cheng, D., Liu, H., Feng, X., Zhou, E., and Liu, Y . Feature interaction interpretability: A case for explain- ing ad-recommendation systems via neural interaction detection.arXiv preprint arXiv:2006.10966,

    2006 arXiv

  2. [2019]

    Modality dropout for improved performance-driven talking faces

    Hussen Abdelaziz, A., Theobald, B.-J., Dixon, P., Knothe, R., Apostoloff, N., and Kajareker, S. Modality dropout for improved performance-driven talking faces. InProceed- ings of the 2020 International Conference on Multimodal Interaction, pp. 378–386,

    2020

  3. [2021]

    P., Ling, C

    Liang, P. P., Ling, C. K., Cheng, Y ., Obolenskiy, A., Liu, Y ., Pandey, R., Wilf, A., Morency, L.-P., and Salakhut- dinov, R. Multimodal learning without labeled multi- modal data: Guarantees and applications.arXiv preprint arXiv:2306.04539, 2023a. Liang, P. P., Ling, C. K., ...

    arXiv

  4. [2023]

    K., Rahman, W., Zadeh, A., Zhong, J., Tanveer, M

    Hasan, M. K., Rahman, W., Zadeh, A., Zhong, J., Tanveer, M. I., Morency, L.-P., et al. Ur-funny: A multimodal language dataset for understanding humor.arXiv preprint arXiv:1904.06618,

    1904 arXiv

  5. [2024]

    and Lizier, J

    Finn, C. and Lizier, J. T. Pointwise partial information decompositionusing the specificity and ambiguity lattices. Entropy, 20(4):297, 2018a. Finn, C. and Lizier, J. T. Probability mass exclusions and the directed components of mutual information.Entropy, 20(11):826, 2018b. F...

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.