{"id":"2c98192f-1e3e-44d9-9211-0a4156353ecf","arxiv_id":"2505.19760","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"A practical guide and benchmark showing that PESQ version and implementation choice can change scores substantially, plus a new repository implementing the latest Corrigendum 2.","lead":"PESQ, a widely used speech quality metric, has many versions and implementations that can produce very different scores. This paper maps those versions, measures the differences, and releases a new open-source tool that implements the latest official corrections.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim that audiolabs/PESQ implements Corrigendum 2 correctly is validated only against the authors' own patched ITU code; a misreading of the corrigendum would be invisible in the reported RMSE=0.01.","rationale":"I read the paper in good faith. The version-comparison results, such as the max difference of 1.82 between P.862 and P.862.2, are computed with the standard ITU reference code and do not depend on the Corrigendum 2 patch; those claims stand independently. The multi-channel section is explicitly inconclusive and is not the main contribution. The single fragile link is the correctness of the Corrigendum 2 implementation, which is the paper's original artifact. Because the validation in Sec. 4.3 compares the new Python package to the authors' own patched version of the same ITU C code, the reported RMSE = 0.01 confirms internal consistency rather than independent correctness. If the corrigendum were misread, both sides of the comparison would carry the same error, and the community would be directed to a subtly wrong metric. The repository being open is helpful and makes independent verification possible, but openness alone does not establish correctness. The reader's weakest_assumption identifies exactly this issue, and I agree with the CONDITIONAL verdict: the paper should not be fully accepted as providing a reliable Corrigendum 2 implementation until the patch is independently verified or released with a documented test suite.","tokens_in":8146,"tokens_out":7979,"duration_ms":83639,"concrete_test":"Have a third party with no access to audiolabs/PESQ independently patch a pristine copy of the ITU-T P.862 reference C code using only ITU-T P.862 Corrigendum 2 (2018), including the corrected filter coefficients and any mapping changes, then run both implementations on the 240 ODAQ signals (downmixed and downsampled as in Sec. 4.1) plus a small set of synthetic wideband stimuli designed to exercise the corrected filter. If RMSE exceeds about 0.02 or max absolute difference exceeds about 0.1, the audiolabs/PESQ implementation is not confirmed to match Corrigendum 2; if differences stay at the ~0.01 level, the self-referential validation concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's main novel contribution is audiolabs/PESQ, claimed to be the first open implementation of P.862.2 with Corrigendum 2. The evidence in Sec. 4.3 for this claim is a comparison between audiolabs/PESQ and 'the ITU reference implementation with the addition of Corrigendum 2,' reporting rho = 1.00 and RMSE = 0.01. But both sides of that comparison were produced by the same authors from the same text-only corrigendum. The ITU code does not contain Corrigendum 2, so a patched C version had to be constructed manually; audiolabs/PESQ is a fork of ludlows/PESQ (which wraps that same ITU C code) with the authors' corrections added. Thus the RMSE = 0.01 verifies that the Python wrapper and the C patch are internally consistent; it does not verify that either one correctly implements the corrigendum. A wrong coefficient, a filter applied at the wrong processing stage, or an incorrect mapping in the corrigendum patch would be present on both sides and would not change the RMSE. The paper does not publish the C patch as a diff or provide independent test vectors, so the central 'reliable access to P.862.2 with Corrigendum 2' claim is not yet independently supported. This does not weaken the version-difference analysis, which uses the unpatched ITU implementation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper surveys the ITU-T P.862 family (P.862, P.862.1, and P.862.2), reviews several openly available implementations, and quantifies score differences on the ODAQ dataset. The headline empirical finding is that version choice matters: comparing P.862.2 without Corrigendum 2 with raw P.862 yields RMSE = 0.62 and a maximum difference of 1.82 on the considered data, while applying Corrigendum 2 changes P.862.2 scores by RMSE = 0.56 and a maximum difference of 1.30. The paper also introduces audiolabs/PESQ, a Python package that implements P.862.2 with Corrigendum 2, and reports that it matches the patched ITU reference implementation with rho = 1.00 and RMSE = 0.01. A final section discusses strategies for applying PESQ to stereo signals and reports that no single strategy dominates on the two speech datasets considered.","tokens_in":8420,"tokens_out":5624,"duration_ms":56299,"significance":"If the Corrigendum 2 implementation is correct, the paper provides a valuable service to the speech-enhancement community: it documents an underappreciated version difference, gives quantitative evidence that researchers should report the exact PESQ version and implementation, and fills a real gap by offering an easy-to-install package for the latest PESQ corrections. The comparative measurements are clearly described, use a public dataset, and the reported RMSE values support the qualitative claim that version differences are large. The paper is also honest about the inconclusive stereo-strategy results. The main weakness is that the central validation of audiolabs/PESQ is self-referential: the reference against which the new package is checked is a patched version of the same ITU C code, patched by the same authors. The contribution is therefore conditional on an independent verification of the corrigendum patch.","major_comments":[{"comment":"The claim that audiolabs/PESQ is a valid out-of-the-box solution for P.862.2 with Corrigendum 2 rests on a comparison against \"the ITU reference implementation with the addition of Corrigendum 2,\" but that reference is not an independent artifact: the corrigendum patch was written by the same authors, and audiolabs/PESQ is a fork of the same ITU C code with the same corrections. An error in interpreting the corrigendum text, such as a wrong filter coefficient or a correction applied at the wrong processing stage, would be present on both sides of the comparison and would not change the reported rho = 1.00 and RMSE = 0.01. To make the central claim load-bearing, the paper should publish the C patch as a reviewable diff against the ITU code and validate the implementation on test vectors derived independently from the corrigendum text or from a separate implementation.","section":"Sec. 4.3, last paragraph"},{"comment":"The headline version-difference statistics, including RMSE = 0.62 and maximum difference 1.82, are computed over the full ODAQ set, which contains non-speech items that the paper itself notes are outside PESQ's application domain. Because the paper's practical recommendation is aimed at speech researchers, the version-difference analysis should also be reported for the speech-only subset of ODAQ. Without that breakdown, it is unclear whether the large differences are representative of typical speech-enhancement evaluations or are inflated by conditions for which PESQ was not designed.","section":"Sec. 4.2, Figs. 2 and 3"}],"minor_comments":[{"comment":"The text says \"audioalabs/PESQ\" but the repository name is \"audiolabs/PESQ\".","section":"Sec. 2.4"},{"comment":"The heading reads \"MA TLAB\" and should read \"MATLAB\".","section":"Sec. 3.3 heading"},{"comment":"Table 1 lists P.862.2 as (11/2005) while reference [4] and the text in Sec. 2.3 use 2007; please clarify the publication and corrigendum dates to avoid confusion.","section":"Table 1 and Sec. 2.3"},{"comment":"The caption calls differences \"negligible\" but the maximum absolute difference is 0.10; \"small\" or \"very small\" would be more precise.","section":"Fig. 4 caption"},{"comment":"The statement that Corrigendum 2 \"is not implemented by any other openly available distribution\" is a strong negative claim supported only by the small set of implementations surveyed here; please qualify it as \"to the best of our knowledge\" or \"among the implementations reviewed in this work.\"","section":"Abstract and Sec. 6"}],"recommendation":"major_revision","confidential_remarks":"The self-referential validation in Sec. 4.3 is the main reason for major revision. The paper's empirical version-difference analysis is sound and useful, but the central novelty claim about audiolabs/PESQ needs an independent check. The paper fits the journal's scope well, and the authors are likely in a good position to provide the missing evidence by releasing the patch and adding independent test vectors."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper delivers two things: a clear map of PESQ versions and implementations, and a new open-source Python package, audiolabs/PESQ (pesqc2), that is the first openly available implementation of P.862.2 with Corrigendum 2. That package fills a real gap—nobody else ships the corrigendum—and the comparative analysis on the ODAQ dataset makes the practical stakes concrete: version choice changes scores by up to 1.82 on the test set, with RMSE 0.62 between raw P.862 and wideband P.862.2. The message that researchers should specify both version and implementation is important and well argued.\n\nThe paper is honest and clearly written. The history section is accurate, the measurements are described in enough detail to reproduce, and the multi-channel section openly reports inconclusive results rather than overclaiming. The use of ODAQ, the authors' own dataset, is a minor note—it is a public, heterogeneous test set, so not a bias problem, but it should be kept in mind.\n\nThe main soft spot is the verification of the new package. In Sec 4.3, audiolabs/PESQ is compared against “the ITU reference implementation with the addition of Corrigendum 2.” The ITU code does not contain the corrigendum; it is a text-only amendment, so the reference was patched by the same authors. The reported RMSE of 0.01 therefore confirms that the Python wrapper and their hand-patched C code agree with each other, not that either is a correct reading of the corrigendum. The paper does not publish the patch as a diff or provide independent test vectors. This is a real limitation, and it also means the with-versus-without corrigendum comparison in Fig 3 rests on the same patch.\n\nThat said, the core findings about version differences between P.862 raw and P.862.2 without the corrigendum do not depend on the patch and are solid. The recommendation to report versions is well supported regardless of whether the corrigendum patch is perfect.\n\nThis paper deserves a serious referee. The practical guidance and the new package are useful to the speech-quality community. I would ask the authors to release the patch and test vectors, or find independent verification, before the Corrigendum 2 implementation is taken at face value. But the paper's core value stands.\n\nVerdict: publish after moderate revision, with the verification gap addressed.","headline":"A practical, useful guide to PESQ versions with a new open implementation of Corrigendum 2; the implementation's validation is self-referential, but the version-comparison findings and the core message are solid.","tokens_in":8957,"tokens_out":3475,"would_cite":true,"duration_ms":32375,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PESQ is not a single metric: version choice can shift scores by up to 1.82 points on the ODAQ testbed, and the paper ships an open implementation of the latest corrected version, P.862.2 with Corrigendum 2.","keywords":["PESQ","speech quality assessment","P.862","P.862.2","Corrigendum 2","MOS-LQO","open-source implementation","ODAQ"],"falsifier":"Have an independent team implement P.862.2 with Corrigendum 2 directly from the ITU text, run it and audiolabs/PESQ on the ODAQ test set, and check whether the RMSE stays near 0.01; a larger gap would refute the claim that the package reliably implements the latest standard.","tokens_in":7959,"feed_emoji":"🎧","tokens_out":6265,"duration_ms":57129,"temperature":0.7,"pith_summary":"The paper argues that “PESQ” is not a single, stable metric: the ITU Recommendation P.862 family comprises narrowband raw scores, narrowband MOS-LQO, and wideband MOS-LQO, plus a 2018 corrigendum that most users never adopted. Using the ODAQ dataset of 240 audio samples, the authors show that choosing one version over another can change scores by up to 1.82 points, and that applying Corrigendum 2 shifts wideband scores by up to 1.30. Because no openly available distribution included the corrigendum, they release audiolabs/PESQ (PyPI package pesqc2), a Python wrapper that implements P.862.2 with Corrigendum 2, and verify it against the ITU reference code patched with the corrigendum at $\\rho=1.00$ and RMSE=0.01. The practical message is that researchers should report the exact PESQ version and implementation, and how multi-channel signals were handled, to keep published scores comparable.","feed_headline":"PESQ versions disagree by up to 1.82 points","feed_subtitle":"Open Python tool for the latest corrected PESQ; version choice can shift scores by 1.82.","key_machinery":"The central objects are the standardized versions of the PESQ algorithm: P.862 (narrowband raw), P.862.1 (mapping to narrowband MOS-LQO), P.862.2 (wideband MOS-LQO), and the 2018 Corrigendum 2, which corrects the wideband input-filter coefficients. The argument is carried by the ITU reference ANSI-C implementation as the comparison anchor, by the ODAQ dataset of 240 audio samples spanning bad to excellent quality, and by the new audiolabs/PESQ package, a fork of ludlows/PESQ that adds the Corrigendum 2 coefficient corrections. The evaluation mechanism is pairwise comparison of score lists using Pearson correlation, RMSE, mean difference, and max absolute difference.","core_discovery":"The authors establish that the many versions of PESQ behave differently enough to matter in practice. On the ODAQ testbed, comparing narrowband raw PESQ (P.862) with wideband MOS-LQO (P.862.2) gives RMSE=0.62 and max difference 1.82; P.862.2 with and without Corrigendum 2 differs by RMSE=0.56 and max 1.30, with the corrigendum correcting a systematic under-prediction of subjective quality. They also compare implementations: ludlows/PESQ matches the ITU reference almost exactly (RMSE=0.03), while audiolabs/torch-pesq deviates by RMSE=0.15 with per-item outliers up to 0.84 because it approximates to stay differentiable. Their contribution is audiolabs/PESQ, which for the first time makes P.862.2 with Corrigendum 2 available as an easy-to-install Python package; against the same-team patched ITU reference it achieves $\\rho=1.00$, RMSE=0.01, mean difference 0.00, and max absolute difference 0.12 on ODAQ. In addition, they document that the ITU reference code handles stereo files by interleaving channels into one long mono signal, which appears unintentional, and that no single alternative strategy (mono downmix versus averaging per-channel scores) is consistently better on the two listening datasets they test.","pith_inferences":["If version differences of this size generalize beyond ODAQ, meta-analyses and leaderboards in speech enhancement that pool PESQ scores from different papers are likely to be confounded; a reporting checklist requiring the exact P.862 variant and implementation would be a cheap remedy.","The corrigendum's stated average under-prediction (0.8) is larger than the mean shift observed here (0.44 on ODAQ), suggesting the correction's impact is content-dependent; a follow-up could measure it on standardized codec and transmission databases to estimate when old scores need recalibration.","Because no stereo strategy consistently wins on the two subjective datasets, dataset maintainers could standardize mono-downmix reporting while optionally publishing per-channel scores, which would let future work recombine them.","The success of a differentiable PESQ approximation suggests a differentiable implementation of the corrigendum-corrected P.862.2 could let training objectives track the latest standard rather than the outdated one."],"forward_implications":["If the central claim is right, any speech-enhancement result that reports only “PESQ” is ambiguous: the same signals can differ by up to 1.82 points solely because of version choice.","The pesqc2 package makes the latest standard (P.862.2 with Corrigendum 2) straightforwardly usable, so the community no longer has to choose between an outdated de facto standard and hand-patching C code.","Because Corrigendum 2 changes wideband scores by up to 1.30 on ODAQ, published results computed without it should be re-examined when compared against newer work.","The documented stereo interleaving behavior of the ITU reference code means multi-channel results from that code are not what users intend; studies should state whether they downmixed or averaged channels.","When precise reporting matters, differentiable approximations like torch-pesq (RMSE 0.15, max 0.84) should be cited as approximations, not as the ITU reference result."],"supporting_citations":[{"why":"ODAQ dataset; supplies the 240 audio samples used for every version and implementation comparison.","marker":"[17]"},{"why":"ITU-T P.862 Corrigendum 2; text defining the filter-coefficient corrections that audiolabs/PESQ implements.","marker":"[5]"},{"why":"ITU-T P.862; defines the original narrowband PESQ algorithm and the reference code used as comparison anchor.","marker":"[2]"},{"why":"ITU-T P.862.2; defines the wideband extension whose corrected and uncorrected versions are compared.","marker":"[4]"},{"why":"ITU-T P.862.1; defines the narrowband MOS-LQO mapping used in the version comparison.","marker":"[3]"}],"fun_headline_variants":["PESQ versions diverge by up to 1.82 points","First open Python PESQ with Corrigendum 2","PESQ score gap: 1.82 points across versions","New PESQ package matches ITU reference closely","PESQ: version choice can flip scores by 1.82"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The validation of the new package compares it against the authors' own manually patched copy of the ITU reference code, so if the Corrigendum 2 text was misapplied, the same error would appear on both sides of the comparison.","fun_headline_variants_meta":{"raw":{"variants":["PESQ versions diverge by up to 1.82 points","First open Python PESQ with Corrigendum 2","PESQ score gap: 1.82 points across versions","New PESQ package matches ITU reference closely","PESQ: version choice can flip scores by 1.82"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000217,"raw_usage":{"total_tokens":1476,"prompt_tokens":1026,"completion_tokens":450,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":642,"completion_tokens_details":{"reasoning_tokens":361}},"tokens_in":642,"tokens_out":450,"duration_ms":4436,"temperature":1.0,"reasoning_tokens":361,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:06:18.657543+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have an independent team implement P.862.2 with Corrigendum 2 directly from the ITU text, run it and audiolabs/PESQ on the ODAQ test set, and check whether the RMSE stays near 0.01; a larger gap would refute the claim that the package reliably implements the latest standard.","supporting_citations":[{"cited_title":"ODAQ: Open Dataset of Audio Quality,","cited_arxiv_id":null,"evidence_quote":"ODAQ dataset; supplies the 240 audio samples used for every version and implementation comparison."},{"cited_title":"Perceptual Evaluation of Speech Quality (PESQ): An Objective Method for End- to-End Speech Quality Assessment of Narrow-Band Tele- phone Networks and Speech Codecs – Corrigendum 2,","cited_arxiv_id":null,"evidence_quote":"ITU-T P.862 Corrigendum 2; text defining the filter-coefficient corrections that audiolabs/PESQ implements."},{"cited_title":"Perceptual Evaluation of Speech Quality (PESQ): An Objective Method for End-to-End Speech Quality Assessment of Narrow-Band Telephone Networks and Speech Codecs,","cited_arxiv_id":null,"evidence_quote":"ITU-T P.862; defines the original narrowband PESQ algorithm and the reference code used as comparison anchor."},{"cited_title":"Wideband Extension to Recommendation P.862 for the Assessment of Wideband Telephone Networks and Speech Codecs,","cited_arxiv_id":null,"evidence_quote":"ITU-T P.862.2; defines the wideband extension whose corrected and uncorrected versions are compared."},{"cited_title":"Mapping Function for Transforming P.862 Raw Results Scores to MOS-LQO,","cited_arxiv_id":null,"evidence_quote":"ITU-T P.862.1; defines the narrowband MOS-LQO mapping used in the version comparison."}],"review_version":1}