REVIEW 3 major objections 5 minor 33 references
fastabx: A library for efficient computation of ABX discriminability
T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read fastabx is a new Python library that computes ABX discriminability scores for arbitrary ON/BY/ACROSS tasks, exactly replicates the outputs of ABXpy, and ran the standard LibriSpeech phoneme ABX task about 65 times faster than ABXpy.
desk verdict A genuinely useful ABX library with a sharp bug report, but the exact-replication claim needs numerical support before I'd trust the headline numbers. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the ABX task, parameterized by three conditions: ON (the attribute shared by A and X but not B), BY (shared by A, B, and X), and ACROSS (shared by A and B but not X). The machinery that carries the argument is a three-class pipeline: Dataset wraps samples and feature access; Task precomputes all valid triples per cell using Polars' lazy operations; Score computes d(x,a) < d(x,b) distances, with DTW alignment implemented as a PyTorch C++/CUDA extension using wavefront parallelism across diagonals. A second load-bearing piece is the frame-slicing convention I = {i in N | t_on <= t_i <= t_off} with t_i = Delta_t/2 + Delta_t i, which defines which feature frames belong to a triphone and is used to judge Libri-Light's slicing as one frame too short.
What would settle it
Run the same ABX task (e.g., LibriSpeech dev-clean, phoneme ON, speaker and context BY) through ABXpy and fastabx and compare per-cell error rates: any numerical difference, beyond floating-point rounding, refutes the exact-replication claim. Alternatively, recompute ZeroSpeech 2021 scores with features sliced to include the extra final frame; if scores are identical to published ones, the claimed Libri-Light bug has no observable effect.
Extended reading notes
Core claim
The discovery is twofold. First, ABX evaluation can be made both general and fast: fastabx uses lazy dataframe operations to precompute task cells and a PyTorch C++/CUDA dynamic-time-warping kernel with wavefront parallelism, reducing the LibriSpeech dev-clean phoneme ABX task from 2 hours 12 minutes to 2 minutes 2 seconds while keeping ABXpy's interface and conventions. Second, the paper identifies an off-by-one error in the Libri-Light evaluation code: features were sliced with features[i_start:i_end] instead of features[i_start:i_end+1], so the last frame of every unit was dropped; the paper argues this follows from the same frame-index convention as ABXpy, and that it matters most for features with large time steps. fastabx replicates ABXpy exactly by default and offers an environment variable to reproduce Libri-Light's shorter slicing.
Load-bearing premise
The paper's strongest claims depend on its assumption that ABXpy's frame-slicing rule, which includes the frame whose time is nearest to both the start and end timestamps, is the correct protocol; if the intended protocol was Libri-Light's shorter slice, then the exact-replication claim, the reported bug, and the speed comparison are all about different tasks.
Editorial extensions
If this is right
- ABXpy users can switch to fastabx as a drop-in replacement, obtaining identical scores in a fraction of the time.
- Past ZeroSpeech 2021 scores computed with Libri-Light were computed with one fewer frame per unit, so their absolute values may be slightly off; the paper does not quantify the effect on published numbers.
- The generic ON/BY/ACROSS interface makes it straightforward to compute ABX scores in new domains or with new condition configurations, such as the context-invariant ABX of prior work that required rewriting Libri-Light.
- The GPU DTW kernel makes large-scale ABX evaluations practical on a single machine, enabling layer-by-layer and contrast-by-contrast analyses.
Reading between the lines
- If the slicing bug is genuine, benchmark comparisons between models evaluated with Libri-Light may need rechecking, especially for representations with coarse temporal resolution (40 or 80 ms steps) where a dropped frame is a larger fraction of the unit.
- A natural stress test is to recompute ZeroSpeech 2021 leaderboard scores with fastabx's default (ABXpy) slicing and compare ranking shifts; the paper does not provide this.
- The same wavefront-parallel DTW approach could be adapted to other alignment-heavy evaluation metrics, such as word error rate or frame-level canonical correlation, where DTW is a bottleneck.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces fastabx, a Python library for computing ABX discriminability scores on generic (ON, BY, ACROSS) tasks. The library provides Dataset, Task, and Score classes, a PyTorch C++/CUDA backend for dynamic time warping, and a command-line interface. The authors claim that fastabx is faster than ABXpy and Libri-Light, exactly replicates ABXpy results, and that Libri-Light's feature slicing is one frame too short, affecting past ZeroSpeech 2021 scores. Two demonstration analyses are included: layer-wise phoneme/speaker ABX for HuBERT and WavLM with and without Spin, and a correlation between ABX error rates and articulatory feature distances.
Significance. If the central claims hold, fastabx is a valuable community resource: it offers a generic, maintainable, and much faster replacement for ABXpy, with a clean interface and GPU support. The alleged Libri-Light slicing bug, if confirmed, would have broad implications for interpreting ZeroSpeech 2021 results. The paper also provides an open-source implementation with tests, which supports reproducibility. However, the exact-replication claim and the bug report currently rest on a convention choice with no numerical evidence, and the timing benchmark lacks sufficient detail. The scientific significance thus depends on verification the manuscript does not currently provide.
major comments (3)
- [Section 3.3] The sentence "Results from ABXpy are exactly replicated by fastabx" is a central claim but is never supported by numerical evidence. I request a direct comparison: report ABX error rates (e.g., on phoneme and on speaker) computed by ABXpy, Libri-Light, fastabx in default mode, and fastabx with the Libri-Light compatibility environment variable, on the same subset (e.g., LibriSpeech dev-clean) using the same input features. Report per-cell or overall differences, maximum absolute deviation, and the fraction of cells where the two libraries give different outcomes. Without such a comparison, the exact-replication claim is unverifiable.
- [Appendix A] The slicing-bug conclusion depends entirely on the inclusive convention in Eq. (1), I = {i in N | t_on <= t_i <= t_off}, with t_i = Delta_t/2 + Delta_t i. Under this convention, Libri-Light's features[i_start:i_end] is indeed one frame short. But the paper does not establish from any independent source that this inclusive, center-aligned convention is the protocol of record for ZeroSpeech/ABXpy. If the intended protocol treats the interval as half-open [t_on, t_off), then Libri-Light is correct, fastabx's default adds one extra frame to every token, and the Table 2 benchmark compares two different ABX tasks rather than two implementations of the same task. Please cite the protocol specification (e.g., ZeroSpeech 2021 documentation or ABXpy source) that fixes the slicing convention, and show that ABXpy's actual slicing code matches Eq. (1). A numerical match between fastabx and published ZeroSpeech 2021 scores would also help.
- [Table 2] The wall-clock timing benchmark is reported from what appears to be a single run, with no error bars or repeated-measurement statistics. The feature extractor used to obtain representations (model, layer, and whether features were pre-extracted with a specific script) is not specified. To make the speed comparison reproducible, please specify the exact feature set, the alignment/item file, the code versions of ABXpy, Libri-Light, and fastabx, and the command lines used. Reporting at least three runs with mean and standard deviation would establish that the 2x speedup over Libri-Light is not noise. Also clarify whether all libraries were evaluated on the same number of triples after the same subsampling procedure (the text says "without subsampling," but the end-to-end pipeline includes triple construction, so the same task definition must be confirmed).
minor comments (5)
- [Section 2.1] The phrase "anything over than speech" appears to be a typo for "anything other than speech".
- [Section 3.2] The DTW recurrence is written for 1 <= i <= N-1 and 1 <= j <= M-1, but the boundary conditions (c_{0,j}, c_{i,0}, and c_{0,0}) are not stated. Please specify how the dynamic programming matrix is initialized.
- [Appendix A] N is used in I = {i in N | ...} without being defined; state whether N is the non-negative integers or the natural numbers starting at 1, and note that t_i = Delta_t/2 + Delta_t i implies frame centers are located at odd multiples of Delta_t/2.
- [Figure 6] The correlation value in the figure appears with a placeholder square ("r = □0.477"). Please render the numeric value cleanly.
- [Section 3.1] The item file format is described as "a format introduced in ABXpy," but no citation or pointer to a format specification is given; adding a reference or link would help users create compatible item files.
Circularity Check
No significant circularity: the library claims are conformance and performance claims against external implementations.
full rationale
The paper's central claims are (1) that fastabx reproduces ABXpy results exactly and (2) that it is faster than ABXpy and Libri-Light. These are empirical compatibility and performance claims, not derived results. The exact-replication claim is anchored to ABXpy, an external implementation, and the slicing discussion in Appendix A explicitly identifies the inclusive frame-index convention as a choice inherited from ABXpy rather than as a theorem. No parameter is fitted to a target and then reported as a prediction, and no load-bearing argument reduces to a self-citation. The Libri-Light 'one frame too short' finding is a checkable claim about an external codebase's slicing behavior under a stated convention; whether that convention is the correct protocol is a correctness or reproducibility concern, not a circularity concern. The benchmark in Table 2 is measured wall-clock time against external tools, and the illustrative analyses in Sections 4.1 and 4.2 are applications rather than derivations of the library itself. No circular step can be exhibited from the paper's equations or citations.
Assumptions & free parameters
assumptions (4)
- domain assumption The ABX frame-slicing convention of ABXpy (Eq. 1, including both endpoints and the Delta_t/2 offset) is the correct protocol; Libri-Light's omission of the last frame is a bug.
- standard math The ABX discriminability formula (Sec. 2, D_C = 1/|C| sum 1_{d(a,x)<d(b,x)} + 1/2 1_{d(a,x)=d(b,x)}) is the accepted metric from Schatz et al., and the library's output is comparable to previous evaluations.
- standard math The DTW recurrence c_{i,j} = d(i,j) + min(c_{i-1,j}, c_{i,j-1}, c_{i-1,j-1}) is the correct alignment cost and is what previous ABX libraries computed.
- domain assumption The single benchmark run on LibriSpeech dev-clean with one V100 GPU and 20 CPU cores is representative of typical ABX evaluation workloads.
Cite this review
Pith. "Pith review of fastabx: A library for efficient computation of ABX discriminability." pith.science (2026). https://pith.science/paper/LER53QYN
@misc{pith2026250502692,
author = {Pith},
title = {Pith review of: fastabx: A library for efficient computation of ABX discriminability},
year = {2026},
howpublished = {\url{https://pith.science/paper/LER53QYN}},
note = {Machine review of arXiv:2505.02692}
}
read the original abstract
We introduce fastabx, a high-performance Python library for building ABX discrimination tasks. ABX is a measure of the separation between generic categories of interest. It has been used extensively to evaluate phonetic discriminability in self-supervised speech representations. However, its broader adoption has been limited by the absence of adequate tools. fastabx addresses this gap by providing a framework capable of constructing any type of ABX task while delivering the efficiency necessary for rapid development cycles, both in task creation and in calculating distances between representations. We believe that fastabx will serve as a valuable resource for the broader representation learning community, enabling researchers to systematically investigate what information can be directly extracted from learned representations across several domains beyond speech processing. The source code is available at https://github.com/bootphon/fastabx.
Figures
Reference graph
Works this paper leans on
-
[1]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. https://proceedings.neurips.cc/paper_files/paper/2020/file/92d1e1eb1cd6f9fba3227870bb6d7f07-Paper.pdf wav2vec 2.0: A framework for self-supervised learning of speech representations . In Advances in Neural Information Processing Systems, volume 33, pages 12449--12460. Curran Associates, Inc
2020
-
[4]
Belviranli, Peng Deng, Laxmi N
Mehmet E. Belviranli, Peng Deng, Laxmi N. Bhuyan, Rajiv Gupta, and Qi Zhu. 2015. https://doi.org/10.1145/2751205.2751243 Peerwave: Exploiting wavefront parallelism on gpus with peer-sm synchronization . In Proceedings of the 29th ACM on International Conference on Supercomputing, ICS '15, page 25–35, New York, NY, USA. Association for Computing Machinery
arXiv 2015
-
[5]
Mar \' a Andrea Cruz Bland \'o n, Nayeli Gonzalez-Gomez, Marvin Lavechin, and Okko R \"a s \"a nen. 2025. https://doi.org/10.1016/j.cognition.2024.106044 Simulating prenatal language exposure in computational models: An exploration study . Cognition, 256:106044
arXiv 2025
-
[6]
Heng-Jui Chang, Alexander H. Liu, and James Glass. 2023. https://doi.org/10.21437/Interspeech.2023-847 Self-supervised fine-tuning for improved content representations by speaker-invariant clustering . In Interspeech 2023, pages 2983--2987
-
[7]
Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, and Furu Wei. 2022. https://doi.org/10.1109/JSTSP.2022.3188113 Wavlm: Large-scale self-supervised pre-training for full stack speech proces...
arXiv 2022
-
[8]
Black, Laurent Besacier, Sakriani Sakti, and Emmanuel Dupoux
Ewan Dunbar, Robin Algayres, Julien Karadayi, Mathieu Bernard, Juan Benjumea, Xuan-Nga Cao, Lucie Miskic, Charlotte Dugrain, Lucas Ondel, Alan W. Black, Laurent Besacier, Sakriani Sakti, and Emmanuel Dupoux. 2019. https://doi.org/10.21437/Interspeech.2019-2904 The zero resource speech challenge 2019: Tts without t . In Interspeech 2019, pages 1088--1092
Show all 33 references
-
[9]
Ewan Dunbar, Xuan Nga Cao, Juan Benjumea, Julien Karadayi, Mathieu Bernard, Laurent Besacier, Xavier Anguera, and Emmanuel Dupoux. 2017. https://doi.org/10.1109/ASRU.2017.8268953 The zero resource speech challenge 2017 . In 2017 IEEE Automatic Speech Recognition and Understand...
2017
-
[10]
Ewan Dunbar, Nicolas Hamilakis, and Emmanuel Dupoux. 2022. https://doi.org/10.1109/JSTSP.2022.3206084 Self-supervised language learning from raw audio: Lessons from the zero resource speech challenge . IEEE Journal of Selected Topics in Signal Processing, 16(6):1211--1226
2022
-
[11]
Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. 2024. https://arxiv.org/abs/2410.00037 Moshi: a speech-text foundation model for real-time dialogue . Preprint, arXiv:2410.00037
2024 arXiv
-
[12]
Mark Hallap, Emmanuel Dupoux, and Ewan Dunbar. 2023. https://doi.org/10.21437/Interspeech.2023-1862 Evaluating context-invariance in unsupervised speech representations . In INTERSPEECH 2023 , pages 2973--2977. ISCA
2023 doi
-
[13]
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. https://doi.org/10.1109/TASLP.2021.3122291 Hubert: Self-supervised speech representation learning by masked prediction of hidden units . IEEE/ACM Transacti...
2021
-
[14]
J. Kahn, M. Rivi \`e re, W. Zheng, E. Kharitonov, Q. Xu, P.E. Mazar \'e , J. Karadayi, V. Liptchinsky, R. Collobert, C. Fuegen, T. Likhomanenko, G. Synnaeve, A. Joulin, A. Mohamed, and E. Dupoux. 2020. https://doi.org/10.1109/ICASSP40776.2020.9052942 Libri- Light : A Benchmark...
2020
-
[15]
Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, and Emmanuel Dupoux. 2021. https://doi.org/10.1162/tacl_a_00430 On generative spoken language modeling from raw audio . Tra...
2021 doi
-
[16]
Marvin Lavechin, Maureen de Seyssel, Hadrien Titeux, Guillaume Wisniewski, Herv \'e Bredin, Alejandrina Cristia, and Emmanuel Dupoux. 2025. https://doi.org/10.1111/desc.13606 Simulating early phonetic and word learning without linguistic categories . Developmental Science, 28(...
2025 doi
-
[17]
Oli Danyi Liu, Hao Tang, and Sharon Goldwater. 2023. https://doi.org/10.21437/Interspeech.2023-871 Self-supervised predictive coding models encode speaker and phonetic information in orthogonal subspaces . In Interspeech 2023, pages 2968--2972
2023 doi
-
[18]
Juliette Millet and Ewan Dunbar. 2022. https://doi.org/10.18653/v1/2022.acl-long.523 Do self-supervised speech models develop human-like perception biases? In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages...
2022 doi
-
[19]
Havtorn, Joakim Edin, Christian Igel, Katrin Kirchhoff, Shang-Wen Li, Karen Livescu, Lars Maaløe, Tara N
Abdelrahman Mohamed, Hung-yi Lee, Lasse Borgholt, Jakob D. Havtorn, Joakim Edin, Christian Igel, Katrin Kirchhoff, Shang-Wen Li, Karen Livescu, Lars Maaløe, Tara N. Sainath, and Shinji Watanabe. 2022. https://doi.org/10.1109/JSTSP.2022.3207050 Self-supervised speech representa...
2022
-
[20]
Mortensen, Patrick Littell, Akash Bharadwaj, Kartik Goyal, Chris Dyer, and Lori Levin
David R. Mortensen, Patrick Littell, Akash Bharadwaj, Kartik Goyal, Chris Dyer, and Lori Levin. 2016. https://aclanthology.org/C16-1328/ P an P hon: A resource for mapping IPA segments to articulatory feature vectors . In Proceedings of COLING 2016, the 26th International Conf...
2016
-
[21]
Tu Anh Nguyen, Maureen de Seyssel, Patricia Rozé, Morgane Rivière, Evgeny Kharitonov, Alexei Baevski, Ewan Dunbar, and Emmanuel Dupoux. 2020. https://arxiv.org/abs/2011.11588 The zero resource speech benchmark 2021: Metrics and baselines for unsupervised spoken language modeli...
2020 arXiv
-
[22]
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. https://doi.org/10.1109/ICASSP.2015.7178964 Librispeech: An asr corpus based on public domain audio books . In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), page...
2015
-
[23]
Ankita Pasad, Chung-Ming Chien, Shane Settle, and Karen Livescu. 2024. https://doi.org/10.1162/tacl_a_00656 What do self-supervised speech models know about words? Transactions of the Association for Computational Linguistics, 12:372--391
2024 doi
-
[24]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu ...
2019
-
[25]
Maxime Poli, Emmanuel Chemla, and Emmanuel Dupoux. 2024 a . https://doi.org/10.18653/v1/2024.emnlp-main.302 Improving spoken language modeling with phoneme classification: A simple fine-tuning approach . In Proceedings of the 2024 Conference on Empirical Methods in Natural Lan...
2024 doi
-
[26]
Maxime Poli, Thomas Schatz, Emmanuel Dupoux, and Marvin Lavechin. 2024 b . https://doi.org/10.34842/y89t-6q31 Modeling the initial state of early phonetic learning in infants . Language Development Research, 5(1)
2024 doi
-
[27]
Thomas Schatz. 2016. https://hal.archives-ouvertes.fr/tel-01407461 ABX-Discriminability Measures and Applications . Theses, Universit \'e Paris 6 (UPMC)
2016
-
[28]
Feldman, Sharon Goldwater, Xuan-Nga Cao, and Emmanuel Dupoux
Thomas Schatz, Naomi H. Feldman, Sharon Goldwater, Xuan-Nga Cao, and Emmanuel Dupoux. 2021. https://doi.org/10.1073/pnas.2001844118 Early phonetic learning without phonetic categories: Insights from large-scale simulations on realistic input . Proceedings of the National Acade...
2021 doi
-
[29]
Thomas Schatz, Vijayaditya Peddinti, Francis Bach, Aren Jansen, Hynek Hermansky, and Emmanuel Dupoux. 2013. https://doi.org/10.21437/Interspeech.2013-441 Evaluating speech features with the minimal-pair abx task: analysis of the classical mfc/plp pipeline . In Interspeech 2013...
2013 doi
-
[30]
Maarten Versteegh, Roland Thiolli \`e re, Thomas Schatz, Xuan Nga Cao, Xavier Anguera, Aren Jansen, and Emmanuel Dupoux. 2015. https://doi.org/10.21437/Interspeech.2015-638 The zero resource speech challenge 2015 . In Interspeech 2015, pages 3169--3173. ISCA
2015 doi
-
[31]
Dan Wells, Hao Tang, and Korin Richmond. 2022. https://doi.org/10.21437/Interspeech.2022-10884 Phonetic analysis of self-supervised representations of english speech . In Interspeech 2022, pages 3583--3587
2022 doi
-
[32]
Lin, Andy T
Shu wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, Tzu-Hsien Huang, Wei-Cheng Tseng, Ko tik Lee, Da-Rong Liu, Zili Huang, Shuyan Dong, Shang-Wen Li, Shinji Watanabe, Abdelrahman Mo...
2021 doi
-
[33]
Salah Zaiem, Youcef Kemiche, Titouan Parcollet, Slim Essid, and Mirco Ravanelli. 2025. https://doi.org/10.1016/j.csl.2024.101695 Speech self-supervised representations benchmarking: A case for larger probing heads . Computer Speech & Language, 89:101695
2025
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.