pith. sign in

arxiv: 2604.15055 · v1 · submitted 2026-04-16 · 📡 eess.SP · cs.SD

Enhancing time-frequency resolution with optimal transport and barycentric fusion of multiple spectrogram

David Valdivia , Elsa Cazelles , C\'edric F\'evotte This is my paper

Pith reviewed 2026-05-10 10:23 UTC · model grok-4.3

classification 📡 eess.SP cs.SD
keywords optimal transportbarycenterspectrogram fusionsuper-resolutiontime-frequency analysisSTFTunbalanced optimal transportsignal processing
0
0 comments X

The pith

Fusing spectrograms with different resolutions via optimal transport barycenters produces a super-resolution time-frequency map that combines their sharpest localizations.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows how to create a higher-resolution spectrogram by treating multiple input spectrograms, each computed with its own STFT window, as points whose average is taken under an optimal transport distance. Because the inputs need not lie on the same time-frequency grid, any choice of analysis parameters can be used and the output grid can be chosen freely by the user. A new transportation cost is introduced that respects the geometry of the time-frequency plane while lowering the computational burden relative to classical Wasserstein barycenters. The computation itself is performed inside the unbalanced optimal transport framework with a block majorization-minimization solver. Experiments on synthetic chirps and recorded speech confirm that the fused result retains the best time localization from one input and the best frequency localization from another, exceeding the performance of an existing unsupervised fusion baseline.

Core claim

The super-resolution spectrogram is defined as the barycenter of the input spectrograms under optimal transport divergences; this construction merges the localization advantages of the inputs without forcing them onto a common grid and is realized efficiently by a new transportation cost together with an unbalanced OT block majorization-minimization algorithm.

What carries the argument

The unbalanced optimal transport barycenter of spectrograms, equipped with a novel transportation cost that preserves time-frequency geometry while lowering complexity.

If this is right

  • Any set of STFT parameters can be used for the input spectrograms; the output is defined on an arbitrary user-chosen grid.
  • The method outperforms an unsupervised state-of-the-art fusion technique on both quantitative metrics and qualitative listening tests for synthetic signals and recorded speech.
  • Several optimal transport divergences can be substituted by changing the transportation cost, with the proposed cost offering a practical trade-off between geometric fidelity and speed.
  • Unbalanced optimal transport allows mass creation and destruction, which accommodates spectrograms whose total energy may differ.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same barycentric construction could be applied to other time-frequency representations such as wavelet scalograms or reassigned spectrograms whose grids also differ.
  • Because the output grid is free, the method could be inserted into iterative pipelines where a coarse map is first computed and then refined on a finer grid without recomputing the original transforms.
  • The framework naturally extends to fusing data from heterogeneous sensors whose sampling rates or analysis windows are mismatched.

Load-bearing premise

That the optimal transport average of spectrograms computed at different resolutions will consistently sharpen both time and frequency features without introducing new distortions that cancel the localization gains.

What would settle it

On a synthetic signal whose true time-frequency support is known exactly, the fused spectrogram either fails to show narrower peaks than the best single input or exhibits spurious energy concentrations not present in any input.

Figures

Figures reproduced from arXiv: 2604.15055 by C\'edric F\'evotte, David Valdivia, Elsa Cazelles.

Figure 1
Figure 1. Figure 1: Impact of window length on spectrogram resolu [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Time-frequency supports of input and output spectro [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Fusion results for a signal consisting of three bass-guitar notes. Amplitude is displayed on a logarithmic scale. (a) [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Signal windowing with windows of different lengths. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Frequency localization error Ef in the single-packet experiment as a function of the tolerance ∆f , averaged over 100 synthetic signals. Spectrogram Et × 10−2 X′ 1 39.0 ± 1.37 X′ 2 2.01 ± 0.25 XG 5.00 ± 0.46 X′ 2.02 ± 0.25 X 2.26 ± 0.27 TABLE I: Temporal localization error Et in the strict case ∆t = 0. Values are reported as mean ± standard error. requires a substantially larger tolerance, around ∆t = 26 m… view at source ↗
Figure 6
Figure 6. Figure 6: Overall concentration error E in the mixture of packets setting in the strict temporal regime ∆t = 0 as a function of the tolerance ∆f , averaged over 100 synthetic signals. For a signal composed of K packets with parameters (f ∗ , t on , t off) = {(f ∗ k , ton k , toff k )} K k=1, we define the mixture’s tolerance support as I∆f ,∆t (f, t on , t off) = [ K k=1 I∆f ,∆t (f ∗ k , ton k , toff k ). (46) We th… view at source ↗
Figure 7
Figure 7. Figure 7: Harmonic concentration error EH on voiced speech segments as a function of the frequency tolerance ∆f , aver￾aged over 100 speech signals from the PTDB-TUG database. Consistently with the harmonic-concentration analysis, both X′ 1 and X exhibit a clear harmonic structure in voiced frames. However, X′ 1 tends to smear energy over time, which reduces the contrast between moments of low and high energy, and a… view at source ↗
Figure 8
Figure 8. Figure 8: Spectrograms computed from a male speech utterance [PITH_FULL_IMAGE:figures/full_fig_p012_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Discrete Fourier Transform of the Hann analysis [PITH_FULL_IMAGE:figures/full_fig_p014_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Time-frequency support with frequency sampling [PITH_FULL_IMAGE:figures/full_fig_p015_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Mel spectrograms computed from the speech excerpt [PITH_FULL_IMAGE:figures/full_fig_p016_11.png] view at source ↗
read the original abstract

Time-frequency representations, such as the short-time Fourier transform (STFT), are fundamental tools for analyzing non-stationary signals. However, their ability to achieve sharp localization in both time and frequency is inherently limited by the Gabor-Heisenberg uncertainty principle. In this paper, we address this limitation by introducing a method to generate super-resolution spectrograms through the fusion of two or more spectrograms with varying resolutions. Specifically, we compute the super-resolution spectrogram as the barycenter of input spectrograms using optimal transport (OT) divergences. Unlike existing fusion approaches, our method does not require the input spectrograms to share the same time-frequency grid. Instead, the input spectrograms can be computed using any STFT parameters, and the resulting super-resolution spectrogram can be defined on an arbitrary user-specified grid. We explore various OT divergences based on different transportation costs. Notably, we introduce a novel transportation cost that preserves time-frequency geometry while significantly reducing computational complexity compared to standard Wasserstein barycenters. We adopt the unbalanced OT framework and derive a new block majorization-minimization algorithm for efficient barycenter computation. We validate the proposed method on controlled synthetic signals and recorded speech using both quantitative and qualitative evaluations. The results show that our approach combines the best localization properties of the input spectrograms and outperforms an unsupervised state-of-the-art fusion method.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript claims to overcome the Gabor-Heisenberg uncertainty limit in time-frequency analysis by fusing two or more spectrograms of differing resolutions into a super-resolution spectrogram defined as their barycenter under unbalanced optimal transport (OT) divergences. Inputs need not share a common grid; the output can be placed on any user-specified grid. A novel transportation cost is introduced that preserves time-frequency geometry at lower complexity than standard Wasserstein distances, an unbalanced OT framework is adopted, and a block majorization-minimization algorithm is derived for efficient computation. Validation is performed on controlled synthetic signals and recorded speech via quantitative metrics and qualitative inspection, with reported outperformance relative to an unsupervised state-of-the-art fusion baseline.

Significance. If the central claim is substantiated, the work offers a principled, grid-flexible alternative to existing spectrogram fusion techniques and could find use in applications requiring high joint time-frequency localization (e.g., audio source separation, radar, or biomedical signal analysis). Credit is due for the algorithmic contribution (block MM solver) and for the empirical demonstration on both synthetic and real data. The approach also avoids the need for explicit grid alignment, which is a practical advantage. Significance is tempered by the absence of a theoretical guarantee that the fused representation improves localization without net fidelity loss or new artifacts.

major comments (2)
  1. [§3] §3 (novel transportation cost and unbalanced OT formulation): the manuscript asserts that the barycenter combines the best localization properties of the inputs, yet no inequality or bound is derived showing that the effective time-frequency spread of the OT barycenter is strictly smaller than that of the best single input. Because the unbalanced formulation permits mass creation/destruction controlled by a free regularization parameter, it is possible for the fusion to trade localization gains for new distortions; this load-bearing point for the super-resolution claim therefore requires either a supporting analysis or an explicit statement that no such guarantee is claimed.
  2. [§5] §5 (quantitative validation): the reported outperformance on synthetic signals is presented without error bars, without details on how data points were excluded or how the unbalanced OT regularization parameter was selected for each trial, and without statistical significance tests against the baseline. These omissions make it impossible to determine whether the observed gains are robust or sensitive to the specific experimental controls, directly affecting confidence in the central empirical claim.
minor comments (2)
  1. [Abstract] Abstract and §1: the phrase 'combines the best localization properties' is used without a preceding definition of the localization metric (e.g., time or frequency spread, or a specific divergence). Adding a short clarifying sentence would improve readability.
  2. [§5] Figure captions (throughout §5): the STFT window lengths, hop sizes, and the exact unbalanced OT regularization value used for each panel should be stated explicitly so that readers can reproduce the visual comparisons.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments. We address each major point below and indicate the planned revisions.

read point-by-point responses

  1. Referee: the manuscript asserts that the barycenter combines the best localization properties of the inputs, yet no inequality or bound is derived showing that the effective time-frequency spread of the OT barycenter is strictly smaller than that of the best single input. Because the unbalanced formulation permits mass creation/destruction controlled by a free regularization parameter, it is possible for the fusion to trade localization gains for new distortions; this load-bearing point for the super-resolution claim therefore requires either a supporting analysis or an explicit statement that no such guarantee is claimed.

    Authors: We agree that no theoretical inequality or bound is derived to prove the OT barycenter has strictly smaller time-frequency spread than the best single input. The claim that the barycenter combines the best localization properties rests on the empirical results shown in Section 5. In the revised manuscript we will add an explicit statement in Section 3 clarifying that no theoretical guarantee of strict improvement is claimed and that the observed super-resolution effect is empirical. We will also discuss the role of the unbalanced regularization parameter and the possibility of fidelity-localization trade-offs. revision: yes

  2. Referee: the reported outperformance on synthetic signals is presented without error bars, without details on how data points were excluded or how the unbalanced OT regularization parameter was selected for each trial, and without statistical significance tests against the baseline. These omissions make it impossible to determine whether the observed gains are robust or sensitive to the specific experimental controls, directly affecting confidence in the central empirical claim.

    Authors: We acknowledge these omissions in the current presentation. In the revised manuscript we will add error bars to all quantitative plots in Section 5, state that no data points were excluded, describe the regularization-parameter selection procedure (a fixed value chosen via preliminary tuning for consistency across trials), and include statistical significance tests (paired t-tests) against the baseline where appropriate. revision: yes

Circularity Check

0 steps flagged

No significant circularity; OT barycenter construction is independent of target resolution claim

full rationale

The paper defines the super-resolution spectrogram explicitly as the unbalanced OT barycenter of input spectrograms (with a new geometry-preserving cost and block MM solver). This is a direct application of established OT theory rather than a self-referential loop. No equation reduces a 'prediction' of enhanced localization to a fitted parameter or prior self-citation by construction. The novel cost is introduced as a computational device, not smuggled via self-citation. Experiments on synthetic and speech data supply independent empirical checks. At most a minor self-citation load (score 2) if prior OT work by the authors is referenced, but it is not load-bearing for the core fusion claim.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 1 invented entities

The central claim rests on standard optimal transport definitions for barycenters and unbalanced variants, plus a novel cost function and custom algorithm introduced for this application; no major free parameters are explicitly fitted in the abstract description.

free parameters (1)
  • unbalanced OT regularization parameter
    Controls balance between transport cost and marginal fidelity in the barycenter computation.
axioms (1)
  • standard math Optimal transport divergences define valid barycenters that minimize aggregate transport costs to input measures
    Core property from optimal transport theory invoked to justify the fusion.
invented entities (1)
  • novel transportation cost preserving time-frequency geometry no independent evidence
    purpose: To maintain structural properties of spectrograms while lowering computational cost compared to standard Wasserstein
    Introduced as a new element in the paper to enable efficient fusion.

pith-pipeline@v0.9.0 · 5547 in / 1482 out tokens · 36696 ms · 2026-05-10T10:23:52.277227+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

2 extracted references · 2 canonical work pages

  1. [1]

    Speech communication, human and machine addison wesley,

    D. O’Shaughnessy, “Speech communication, human and machine addison wesley,”Reading MA, vol. 40, p. 150, 1987

    work page 1987

  2. [2]

    Librosa: Audio and music signal analysis in Python,

    B. McFee et al., “Librosa: Audio and music signal analysis in Python,”SciPy, vol. 2015, no. 18-24, p. 7, 2015

    work page 2015