Pith. sign in

REVIEW 2 major objections 5 minor 21 references

Navigating PESQ: Up-to-Date Versions and Open Implementations

T0 review · 2 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read PESQ is not a single metric: version choice can shift scores by up to 1.82 points on the ODAQ testbed, and the paper ships an open implementation of the latest corrected version, P.862.2 with Corrigendum 2.

desk verdict A practical, useful guide to PESQ versions with a new open implementation of Corrigendum 2; the implementation's validation is self-referential, but the version-comparison findings and the core message are solid. read the letter →

arxiv 2505.19760 v2 pith:UBB65LV2 submitted 2025-05-26 eess.AS

Matteo Torcoli , Mhd Modar Halimeh , Emanuël A. P. Habets This is my paper · ORCID
classification eess.AS
keywords PESQspeechqualityassessmentP.862P.862.2Corrigendum2MOS-LQOopen-sourceimplementationODAQ
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that “PESQ” is not a single, stable metric: the ITU Recommendation P.862 family comprises narrowband raw scores, narrowband MOS-LQO, and wideband MOS-LQO, plus a 2018 corrigendum that most users never adopted. Using the ODAQ dataset of 240 audio samples, the authors show that choosing one version over another can change scores by up to 1.82 points, and that applying Corrigendum 2 shifts wideband scores by up to 1.30. Because no openly available distribution included the corrigendum, they release audiolabs/PESQ (PyPI package pesqc2), a Python wrapper that implements P.862.2 with Corrigendum 2, and verify it against the ITU reference code patched with the corrigendum at $\rho=1.00$ and RMSE=0.01. The practical message is that researchers should report the exact PESQ version and implementation, and how multi-channel signals were handled, to keep published scores comparable.

What carries the argument

The central objects are the standardized versions of the PESQ algorithm: P.862 (narrowband raw), P.862.1 (mapping to narrowband MOS-LQO), P.862.2 (wideband MOS-LQO), and the 2018 Corrigendum 2, which corrects the wideband input-filter coefficients. The argument is carried by the ITU reference ANSI-C implementation as the comparison anchor, by the ODAQ dataset of 240 audio samples spanning bad to excellent quality, and by the new audiolabs/PESQ package, a fork of ludlows/PESQ that adds the Corrigendum 2 coefficient corrections. The evaluation mechanism is pairwise comparison of score lists using Pearson correlation, RMSE, mean difference, and max absolute difference.

What would settle it

Have an independent team implement P.862.2 with Corrigendum 2 directly from the ITU text, run it and audiolabs/PESQ on the ODAQ test set, and check whether the RMSE stays near 0.01; a larger gap would refute the claim that the package reliably implements the latest standard.

Watch

Extended reading notes

Core claim

The authors establish that the many versions of PESQ behave differently enough to matter in practice. On the ODAQ testbed, comparing narrowband raw PESQ (P.862) with wideband MOS-LQO (P.862.2) gives RMSE=0.62 and max difference 1.82; P.862.2 with and without Corrigendum 2 differs by RMSE=0.56 and max 1.30, with the corrigendum correcting a systematic under-prediction of subjective quality. They also compare implementations: ludlows/PESQ matches the ITU reference almost exactly (RMSE=0.03), while audiolabs/torch-pesq deviates by RMSE=0.15 with per-item outliers up to 0.84 because it approximates to stay differentiable. Their contribution is audiolabs/PESQ, which for the first time makes P.862.2 with Corrigendum 2 available as an easy-to-install Python package; against the same-team patched ITU reference it achieves $\rho=1.00$, RMSE=0.01, mean difference 0.00, and max absolute difference 0.12 on ODAQ. In addition, they document that the ITU reference code handles stereo files by interleaving channels into one long mono signal, which appears unintentional, and that no single alternative strategy (mono downmix versus averaging per-channel scores) is consistently better on the two listening datasets they test.

Load-bearing premise

The validation of the new package compares it against the authors' own manually patched copy of the ITU reference code, so if the Corrigendum 2 text was misapplied, the same error would appear on both sides of the comparison.

Editorial extensions

If this is right

  • If the central claim is right, any speech-enhancement result that reports only “PESQ” is ambiguous: the same signals can differ by up to 1.82 points solely because of version choice.
  • The pesqc2 package makes the latest standard (P.862.2 with Corrigendum 2) straightforwardly usable, so the community no longer has to choose between an outdated de facto standard and hand-patching C code.
  • Because Corrigendum 2 changes wideband scores by up to 1.30 on ODAQ, published results computed without it should be re-examined when compared against newer work.
  • The documented stereo interleaving behavior of the ITU reference code means multi-channel results from that code are not what users intend; studies should state whether they downmixed or averaged channels.
  • When precise reporting matters, differentiable approximations like torch-pesq (RMSE 0.15, max 0.84) should be cited as approximations, not as the ITU reference result.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If version differences of this size generalize beyond ODAQ, meta-analyses and leaderboards in speech enhancement that pool PESQ scores from different papers are likely to be confounded; a reporting checklist requiring the exact P.862 variant and implementation would be a cheap remedy.
  • The corrigendum's stated average under-prediction (0.8) is larger than the mean shift observed here (0.44 on ODAQ), suggesting the correction's impact is content-dependent; a follow-up could measure it on standardized codec and transmission databases to estimate when old scores need recalibration.
  • Because no stereo strategy consistently wins on the two subjective datasets, dataset maintainers could standardize mono-downmix reporting while optionally publishing per-channel scores, which would let future work recombine them.
  • The success of a differentiable PESQ approximation suggests a differentiable implementation of the corrigendum-corrected P.862.2 could let training objectives track the latest standard rather than the outdated one.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. This paper surveys the ITU-T P.862 family (P.862, P.862.1, and P.862.2), reviews several openly available implementations, and quantifies score differences on the ODAQ dataset. The headline empirical finding is that version choice matters: comparing P.862.2 without Corrigendum 2 with raw P.862 yields RMSE = 0.62 and a maximum difference of 1.82 on the considered data, while applying Corrigendum 2 changes P.862.2 scores by RMSE = 0.56 and a maximum difference of 1.30. The paper also introduces audiolabs/PESQ, a Python package that implements P.862.2 with Corrigendum 2, and reports that it matches the patched ITU reference implementation with rho = 1.00 and RMSE = 0.01. A final section discusses strategies for applying PESQ to stereo signals and reports that no single strategy dominates on the two speech datasets considered.

Significance. If the Corrigendum 2 implementation is correct, the paper provides a valuable service to the speech-enhancement community: it documents an underappreciated version difference, gives quantitative evidence that researchers should report the exact PESQ version and implementation, and fills a real gap by offering an easy-to-install package for the latest PESQ corrections. The comparative measurements are clearly described, use a public dataset, and the reported RMSE values support the qualitative claim that version differences are large. The paper is also honest about the inconclusive stereo-strategy results. The main weakness is that the central validation of audiolabs/PESQ is self-referential: the reference against which the new package is checked is a patched version of the same ITU C code, patched by the same authors. The contribution is therefore conditional on an independent verification of the corrigendum patch.

major comments (2)
  1. [Sec. 4.3, last paragraph] The claim that audiolabs/PESQ is a valid out-of-the-box solution for P.862.2 with Corrigendum 2 rests on a comparison against "the ITU reference implementation with the addition of Corrigendum 2," but that reference is not an independent artifact: the corrigendum patch was written by the same authors, and audiolabs/PESQ is a fork of the same ITU C code with the same corrections. An error in interpreting the corrigendum text, such as a wrong filter coefficient or a correction applied at the wrong processing stage, would be present on both sides of the comparison and would not change the reported rho = 1.00 and RMSE = 0.01. To make the central claim load-bearing, the paper should publish the C patch as a reviewable diff against the ITU code and validate the implementation on test vectors derived independently from the corrigendum text or from a separate implementation.
  2. [Sec. 4.2, Figs. 2 and 3] The headline version-difference statistics, including RMSE = 0.62 and maximum difference 1.82, are computed over the full ODAQ set, which contains non-speech items that the paper itself notes are outside PESQ's application domain. Because the paper's practical recommendation is aimed at speech researchers, the version-difference analysis should also be reported for the speech-only subset of ODAQ. Without that breakdown, it is unclear whether the large differences are representative of typical speech-enhancement evaluations or are inflated by conditions for which PESQ was not designed.
minor comments (5)
  1. [Sec. 2.4] The text says "audioalabs/PESQ" but the repository name is "audiolabs/PESQ".
  2. [Sec. 3.3 heading] The heading reads "MA TLAB" and should read "MATLAB".
  3. [Table 1 and Sec. 2.3] Table 1 lists P.862.2 as (11/2005) while reference [4] and the text in Sec. 2.3 use 2007; please clarify the publication and corrigendum dates to avoid confusion.
  4. [Fig. 4 caption] The caption calls differences "negligible" but the maximum absolute difference is 0.10; "small" or "very small" would be more precise.
  5. [Abstract and Sec. 6] The statement that Corrigendum 2 "is not implemented by any other openly available distribution" is a strong negative claim supported only by the small set of implementations surveyed here; please qualify it as "to the best of our knowledge" or "among the implementations reviewed in this work."

Circularity Check

1 steps flagged · score 2.0 of 10

P.862.2 Corrigendum 2 validation uses a same-team patched reference, but no fitted inputs or self-citation chains drive the main comparisons.

  1. other [Sec. 4.3 (Comparing implementations), final paragraph; cf. Sec. 3.1 and Sec. 3.2]
    "our audiolabs/PESQ was compared against the ITU reference implementation with the addition of Corrigendum 2. The differences are negligible (hence, plot not included), with ρ = 1.00, RMSE = 0.01, mean difference = 0.00, and max absolute difference = 0.12. This confirms that audiolabs/PESQ is a valid out-of-the-box solution for computing P.862.2 with Corrigendum 2."

    The reference side is the authors' own patch: Sec. 3.1 says Corrigendum 2 'specifies these changes in text form, and the actual corrections are left to be implemented by the users of the reference code,' and Sec. 3.2 says audiolabs/PESQ was created 'by forking from ludlows/PESQ and adding the corrections detailed in Corrigendum 2.' The 'ITU reference implementation with the addition of Corrigendum 2' is thus the same team's C code built from the same text-only corrigendum, and audiolabs/PESQ is the Python wrapper around that patch. RMSE = 0.01 verifies wrapper/patch consistency, not independent correctness; a wrong coefficient would be identical on both sides.

full rationale

No significant circularity beyond the self-constructed validation benchmark. The main empirical results comparing PESQ versions (Sec. 4.2) use the ITU reference code and the ODAQ dataset, and the implementation comparisons of ludlows/PESQ and torch-pesq against the ITU reference (Figs. 4-5) are external checks. The claim that audiolabs/PESQ 'reliably' implements Corrigendum 2 is specifically supported only by agreement with the authors' own patched ITU code, which makes the benchmark self-referential and weakens independent verification, but no parameter is fit to the target and no conclusion is logically forced by definition. Therefore the overall circularity score is low (2/10). If the C patch were published as a diff or tested against independent vectors, this residual concern would disappear.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No free parameters are fitted; the paper makes descriptive comparisons. The main assumptions are the representativeness of the ODAQ testbed, the correctness of the ITU reference code, the authors' interpretation of Corrigendum 2, and the validity of the listening test scores used for stereo strategy evaluation.

assumptions (5)
  • domain assumption The ODAQ dataset, including non-speech audio items, is a suitable and representative testbed for quantifying PESQ version and implementation differences.
    Sec. 4.1 selects ODAQ for the comparison; the conclusions about RMSE and max differences are drawn from this dataset and assumed to generalize.
  • domain assumption The ITU reference ANSI-C implementation correctly computes P.862, P.862.1, and P.862.2 without Corrigendum 2, and serves as ground truth.
    Sec. 3.1 and Sec. 4 treat the ITU reference code as the comparison point for all other implementations.
  • domain assumption The text of ITU-T P.862 Corrigendum 2 can be unambiguously translated into filter coefficient changes by the authors.
    Sec. 4.3 validates audiolabs/PESQ against the ITU reference code with the authors' own application of Corrigendum 2; a misreading of the corrigendum would make the small RMSE meaningless.
  • domain assumption Resampling with librosa's soxr_hq and passive downmixing preserve the audio characteristics relevant to PESQ differences.
    Sec. 4.1 describes the preprocessing; incorrect resampling behavior could alter PESQ inputs, though the effect is bounded.
  • domain assumption The MUSHRA listening test scores from references [20] and [21] are valid ground truth for comparing stereo strategies.
    Sec. 5.3 uses these subjective scores to evaluate MONO DMX, AVG SCORES, and STEREO strategies; if the ground truth is flawed, the comparison of strategies is unreliable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Navigating PESQ: Up-to-Date Versions and Open Implementations." pith.science (2026). https://pith.science/paper/UBB65LV2

@misc{pith2026250519760,
  author       = {Pith},
  title        = {Pith review of: Navigating PESQ: Up-to-Date Versions and Open Implementations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UBB65LV2}},
  note         = {Machine review of arXiv:2505.19760}
}
read the original abstract

Perceptual Evaluation of Speech Quality (PESQ) is an objective quality measure that remains widely used despite its withdrawal by the International Telecommunication Union (ITU). PESQ has evolved over two decades, with multiple versions and publicly available implementations emerging during this time. Different versions and their updates can be overwhelming, especially for new PESQ users. This work provides practical guidance on the different versions and implementations of PESQ. We show that differences can be significant, especially between PESQ versions. We stress the importance of specifying the exact version and implementation that is used to compute PESQ, and possibly to detail how multi-channel signals are handled. These practices would facilitate the interpretation of results and allow comparisons of PESQ scores between different studies. We also provide a repository that implements the latest corrections to PESQ, i.e., Corrigendum 2, which is not implemented by any other openly available distribution: https://github.com/audiolabs/PESQ.

Figures

Figures reproduced from arXiv: 2505.19760 by the authors.

Figure 1
Figure 1. Mapping from narrowband raw PESQ (P.862) [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Raw PESQ scores (P.862) compared with wide [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. PESQ P.862.2 with and without Corrigendum 2. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (2 more)
Figure 6
Figure 6. Figure 6: Comparing strategies for using PESQ (P.862.2 [PITH_FULL_IMAGE:figures/full_fig_p004_6.png]
Figure 5
Figure 5. Figure 5: audiolabs/torch-pesq compared with the ITU ref [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 21 canonical work pages

  1. [1]

    Perceptual Evaluation of Speech Quality (PESQ) - A new Method for Speech Quality Assessment of Telephone Networks and Codecs,

    A. W. Rix, J. G. Beerends, M. P. Hollier, and A. P. Hek- stra, “Perceptual Evaluation of Speech Quality (PESQ) - A new Method for Speech Quality Assessment of Telephone Networks and Codecs,” in IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP) , pp. 749–752, 2001

    work page 2001

  2. [2]

    Perceptual Evaluation of Speech Quality (PESQ): An Objective Method for End-to-End Speech Quality Assessment of Narrow-Band Telephone Networks and Speech Codecs,

    ITU-T P.862, “Perceptual Evaluation of Speech Quality (PESQ): An Objective Method for End-to-End Speech Quality Assessment of Narrow-Band Telephone Networks and Speech Codecs,” 2001. and subsequent amendments and corrigenda

    work page 2001

  3. [3]

    Mapping Function for Transforming P.862 Raw Results Scores to MOS-LQO,

    ITU-T P.862.1, “Mapping Function for Transforming P.862 Raw Results Scores to MOS-LQO,” 2003

    work page 2003

  4. [4]

    Wideband Extension to Recommendation P.862 for the Assessment of Wideband Telephone Networks and Speech Codecs,

    ITU-T P.862.2, “Wideband Extension to Recommendation P.862 for the Assessment of Wideband Telephone Networks and Speech Codecs,” 2007. and subsequent corrigendum

    work page 2007

  5. [5]

    Perceptual Evaluation of Speech Quality (PESQ): An Objective Method for End- to-End Speech Quality Assessment of Narrow-Band Tele- phone Networks and Speech Codecs – Corrigendum 2,

    ITU-T P.862 Corrigendum 2, “Perceptual Evaluation of Speech Quality (PESQ): An Objective Method for End- to-End Speech Quality Assessment of Narrow-Band Tele- phone Networks and Speech Codecs – Corrigendum 2,” 2018

    work page 2018

  6. [6]

    Perceptual Objective Listening Quality Assessment (POLQA), The Third Gener- ation ITU-T Standard for End-to-End Speech Quality Mea- surement part I - Temporal Alignment,

    J. G. Beerends, C. Schmidmer, J. Berger, M. Obermann, R. Ullmann, J. Pomy, and M. Keyhl, “Perceptual Objective Listening Quality Assessment (POLQA), The Third Gener- ation ITU-T Standard for End-to-End Speech Quality Mea- surement part I - Temporal Alignment,” Journal of the Au- dio Engineering Society, vol. 61, no. 6, pp. 366–384, 2013

    work page 2013

  7. [7]

    Perceptual Objective Listening Quality Pre- diction,

    ITU-T P.863, “Perceptual Objective Listening Quality Pre- diction,” 2018. and accompanying documents

    work page 2018

  8. [8]

    MetricGAN+: An Improved Ver- sion of MetricGAN for Speech Enhancement,

    S.-W. Fu, C. Yu, T.-A. Hsieh, P. Plantinga, M. Ravanelli, X. Lu, and Y . Tsao, “MetricGAN+: An Improved Ver- sion of MetricGAN for Speech Enhancement,” in INTER- SPEECH, pp. 201–205, 2021

    work page 2021

Show all 21 references
  1. [9]

    Towards Efficient Models for Real-Time Deep Noise Suppression,

    S. Braun, H. Gamper, C. K. Reddy, and I. Tashev, “Towards Efficient Models for Real-Time Deep Noise Suppression,” in IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pp. 656–660, 2021

    2021

  2. [10]

    Deep Noise Suppres- sion Maximizing Non-Differentiable PESQ Mediated by a Non-Intrusive PESQNet,

    Z. Xu, M. Strake, and T. Fingscheidt, “Deep Noise Suppres- sion Maximizing Non-Differentiable PESQ Mediated by a Non-Intrusive PESQNet,” IEEE/ACM Transactions on Au- dio, Speech, and Language Processing , vol. 30, pp. 1572– 1585, 2022

    2022

  3. [11]

    The PESQetarian: On the Relevance of Goodhart’s Law for Speech Enhancement,

    D. de Oliveira, S. Welker, J. Richter, and T. Gerkmann, “The PESQetarian: On the Relevance of Goodhart’s Law for Speech Enhancement,” in INTERSPEECH, 2024

    2024

  4. [12]

    Wawenets: A No- Reference Convolutional Waveform-Based Approach to Es- timating Narrowband and Wideband Speech Quality,

    A. A. Catellier and S. D. V oran, “Wawenets: A No- Reference Convolutional Waveform-Based Approach to Es- timating Narrowband and Wideband Speech Quality,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 331–335, 2020

    2020

  5. [13]

    Torchaudio-Squim: Reference-Less Speech Quality and Intelligibility Measures in Torchaudio,

    A. Kumar, K. Tan, Z. Ni, P. Manocha, X. Zhang, E. Hen- derson, and B. Xu, “Torchaudio-Squim: Reference-Less Speech Quality and Intelligibility Measures in Torchaudio,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5, 2023

    2023

  6. [14]

    Objective Measures of Perceptual Audio Quality Reviewed: An Evaluation of Their Application Domain Dependence,

    M. Torcoli, T. Kastner, and J. Herre, “Objective Measures of Perceptual Audio Quality Reviewed: An Evaluation of Their Application Domain Dependence,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 29, pp. 1530–1541, 2021

    2021

  7. [15]

    Mean Opinion Score (MOS) Terminol- ogy,

    ITU-T P.800.1, “Mean Opinion Score (MOS) Terminol- ogy,” 2016

    2016

  8. [16]

    P. C. Loizou, Speech Enhancement: Theory and Practice (1st Edition). Taylor & Francis, CRC Press, 2007

    2007

  9. [17]

    ODAQ: Open Dataset of Audio Quality,

    M. Torcoli, C.-W. Wu, S. Dick, P. A. Williams, M. M. Halimeh, W. Wolcott, and E. A. Habets, “ODAQ: Open Dataset of Audio Quality,” in IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), pp. 836–840, 2024

    2024

  10. [18]

    librosa: Audio and Music Signal Analysis in Python,

    B. McFee, C. Raffel, D. Liang, D. P. Ellis, M. McVicar, E. Battenberg, and O. Nieto, “librosa: Audio and Music Signal Analysis in Python,” in 14th Python in Science Con- ference, pp. 18–25, 2015

    2015

  11. [19]

    Recommendation ITU-R BS.1534-3 Method for the subjective assessment of intermediate qual- ity level of audio systems,

    ITU-R BS.1534-3, “Recommendation ITU-R BS.1534-3 Method for the subjective assessment of intermediate qual- ity level of audio systems,” 2015

    2015

  12. [20]

    Expanding and An- alyzing ODAQ – the Open Dataset of Audio Quality,

    S. Dick, C. Thompson, C.-W. Wu, M. Torcoli, P. Delgado, P. A. Williams, , and E. A. P. Habets, “Expanding and An- alyzing ODAQ – the Open Dataset of Audio Quality,” in 157th Audio Engineering Society Convention , 2024

    2024

  13. [21]

    A Hands- on Comparison of DNNs for Dialog Separation Using Transfer Learning from Music Source Separation,

    M. Strauss, J. Paulus, M. Torcoli, and B. Edler, “A Hands- on Comparison of DNNs for Dialog Separation Using Transfer Learning from Music Source Separation,” in IN- TERSPEECH, pp. 3900–3904, 2021

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.