{"id":"ca0f9b57-fdd4-45d7-9715-796a2af85a4b","arxiv_id":"2505.02692","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper introduces fastabx, a fast generic ABX discrimination library, and reports that the Libri-Light evaluation code sliced speech features one frame too short at the end.","lead":"fastabx is a new open-source Python library that computes ABX discrimination scores, a zero-resource measure of how well categories are separated in learned representations. It claims to match the flexibility of the old ABXpy tool while running about 65 times faster, and it exposes a one-frame slicing bug in the widely used Libri-Light evaluation code.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Exact replication and the Libri-Light slicing bug rest entirely on Appendix A's inclusive convention, with no numerical score comparison; if the official protocol is half-open, every fastabx default score changes and Table 2 compares different tasks.","rationale":"The reader's weakest assumption correctly identifies the load-bearing point: the exact-replication and Libri-Light-bug claims depend on ABXpy's inclusive slicing convention (Eq. 1) being the protocol of record. I agree because a one-frame shift at the end of every segment changes every distance, so the speed comparison and the 'bug' claim are not independent of this convention. The absence of numerical ABXpy-versus-fastabx score comparison means the paper does not currently demonstrate the central empirical claim; the open repository is real evidence and makes the proposed check feasible, so this is an addressable condition rather than a fatal flaw. I keep the reader's CONDITIONAL verdict: the concern is concrete and load-bearing, but a single reproducible comparison against the official ZeroSpeech 2021 code would settle it.","tokens_in":9476,"tokens_out":8625,"duration_ms":98792,"concrete_test":"On a fixed LibriSpeech dev-clean subset with one named feature extractor (e.g., wav2vec2 layer 6, 20 ms stride), the item file used by ZeroSpeech 2021, and the corresponding alignments, run the official Libri-Light evaluation and fastabx in both default and Libri-Light-compatible modes, then compare the per-token frame indices and per-cell ABX error rates. The version whose frame indices match Eq. (1) and whose error rates reproduce the published ZeroSpeech 2021 dev-clean scores identifies the protocol of record, settling whether the Libri-Light slice is genuinely one frame short.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's strongest claim, 'Results from ABXpy are exactly replicated by fastabx' (Sec. 3.3), is supported only by a stated convention choice, not by any numerical comparison. Appendix A defines the selected indices as I = {i in N | t_on <= t_i <= t_off}, with t_i = Delta_t/2 + Delta_t i, and concludes that Libri-Light's features[i_start:i_end] 'was always one frame too short at the end.' This conclusion is valid only if the ZeroSpeech/ABXpy protocol of record is this inclusive, center-aligned frame definition. If the intended protocol is the half-open interval [t_on, t_off), then Libri-Light's slice is correct, fastabx's default i_end+1 adds one extra frame to every token, the reported Libri-Light bug is not a bug, and the Table 2 speed comparison is between two different ABX tasks rather than two implementations of the same task. No independent protocol specification is cited, and no ABX error rates from ABXpy, Libri-Light, or fastabx are reported to corroborate exact replication.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces fastabx, a Python library for computing ABX discriminability scores on generic (ON, BY, ACROSS) tasks. The library provides Dataset, Task, and Score classes, a PyTorch C++/CUDA backend for dynamic time warping, and a command-line interface. The authors claim that fastabx is faster than ABXpy and Libri-Light, exactly replicates ABXpy results, and that Libri-Light's feature slicing is one frame too short, affecting past ZeroSpeech 2021 scores. Two demonstration analyses are included: layer-wise phoneme/speaker ABX for HuBERT and WavLM with and without Spin, and a correlation between ABX error rates and articulatory feature distances.","tokens_in":9697,"tokens_out":3902,"duration_ms":41932,"significance":"If the central claims hold, fastabx is a valuable community resource: it offers a generic, maintainable, and much faster replacement for ABXpy, with a clean interface and GPU support. The alleged Libri-Light slicing bug, if confirmed, would have broad implications for interpreting ZeroSpeech 2021 results. The paper also provides an open-source implementation with tests, which supports reproducibility. However, the exact-replication claim and the bug report currently rest on a convention choice with no numerical evidence, and the timing benchmark lacks sufficient detail. The scientific significance thus depends on verification the manuscript does not currently provide.","major_comments":[{"comment":"The sentence \"Results from ABXpy are exactly replicated by fastabx\" is a central claim but is never supported by numerical evidence. I request a direct comparison: report ABX error rates (e.g., on phoneme and on speaker) computed by ABXpy, Libri-Light, fastabx in default mode, and fastabx with the Libri-Light compatibility environment variable, on the same subset (e.g., LibriSpeech dev-clean) using the same input features. Report per-cell or overall differences, maximum absolute deviation, and the fraction of cells where the two libraries give different outcomes. Without such a comparison, the exact-replication claim is unverifiable.","section":"Section 3.3"},{"comment":"The slicing-bug conclusion depends entirely on the inclusive convention in Eq. (1), I = {i in N | t_on <= t_i <= t_off}, with t_i = Delta_t/2 + Delta_t i. Under this convention, Libri-Light's features[i_start:i_end] is indeed one frame short. But the paper does not establish from any independent source that this inclusive, center-aligned convention is the protocol of record for ZeroSpeech/ABXpy. If the intended protocol treats the interval as half-open [t_on, t_off), then Libri-Light is correct, fastabx's default adds one extra frame to every token, and the Table 2 benchmark compares two different ABX tasks rather than two implementations of the same task. Please cite the protocol specification (e.g., ZeroSpeech 2021 documentation or ABXpy source) that fixes the slicing convention, and show that ABXpy's actual slicing code matches Eq. (1). A numerical match between fastabx and published ZeroSpeech 2021 scores would also help.","section":"Appendix A"},{"comment":"The wall-clock timing benchmark is reported from what appears to be a single run, with no error bars or repeated-measurement statistics. The feature extractor used to obtain representations (model, layer, and whether features were pre-extracted with a specific script) is not specified. To make the speed comparison reproducible, please specify the exact feature set, the alignment/item file, the code versions of ABXpy, Libri-Light, and fastabx, and the command lines used. Reporting at least three runs with mean and standard deviation would establish that the 2x speedup over Libri-Light is not noise. Also clarify whether all libraries were evaluated on the same number of triples after the same subsampling procedure (the text says \"without subsampling,\" but the end-to-end pipeline includes triple construction, so the same task definition must be confirmed).","section":"Table 2"}],"minor_comments":[{"comment":"The phrase \"anything over than speech\" appears to be a typo for \"anything other than speech\".","section":"Section 2.1"},{"comment":"The DTW recurrence is written for 1 <= i <= N-1 and 1 <= j <= M-1, but the boundary conditions (c_{0,j}, c_{i,0}, and c_{0,0}) are not stated. Please specify how the dynamic programming matrix is initialized.","section":"Section 3.2"},{"comment":"N is used in I = {i in N | ...} without being defined; state whether N is the non-negative integers or the natural numbers starting at 1, and note that t_i = Delta_t/2 + Delta_t i implies frame centers are located at odd multiples of Delta_t/2.","section":"Appendix A"},{"comment":"The correlation value in the figure appears with a placeholder square (\"r = □0.477\"). Please render the numeric value cleanly.","section":"Figure 6"},{"comment":"The item file format is described as \"a format introduced in ABXpy,\" but no citation or pointer to a format specification is given; adding a reference or link would help users create compatible item files.","section":"Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"This is a software/system paper whose acceptance ultimately hinges on whether the exact-replication and slicing-bug claims hold. The authors are well-placed within the ZeroSpeech/ABXpy lineage, which makes an independent, externally visible protocol citation and a numerical score comparison all the more important. The paper also does not include any new scientific results beyond illustrative examples, so the significance rests almost entirely on the tool and the bug report. I would encourage the editor to require the numerical verification before publication; the scope of the paper can accommodate this without a redesign."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this about fastabx: it is a real, usable library that does what it says—generic ABX task construction plus GPU-accelerated DTW—and the paper contains one genuinely interesting finding, namely that Libri-Light's feature slicing drops the final frame. That bug report (Appendix A) is the most valuable part of the paper. It is clearly derived, it states the convention explicitly, and it is externally checkable against the Libri-Light codebase. That alone makes the paper worth a serious look.\n\nThe library design is sensible. The ON/BY/ACROSS abstraction from ABXpy is carried over cleanly, the Polars-based task construction is a real speed-up over the old triple-loop, and the wavefront DTW parallelization is a legitimate engineering contribution. The paper is honest about the current CUDA utilization being suboptimal, and the code is MIT-licensed with a public repo. These are marks in its favor.\n\nThe soft spots are real but mostly fixable. The claim that \"Results from ABXpy are exactly replicated\" (Sec. 3.3) is supported only by a stated convention choice, not by any reported error rates. The paper should show a numerical comparison: fastabx versus ABXpy on a small subset, with the score to several decimals. Without that, the strongest claim is an assertion. The timing benchmark in Table 2 is a single run with no error bars, and the feature extractor is not named—so I can't tell whether the 2-minute wall time applies to MFCC, HuBERT, or something else. Also, there is no commit hash or environment spec, which makes the benchmark hard to reproduce.\n\nOn the stress-test concern: I think the worry about the inclusivity convention is slightly overstated but worth flagging. The paper openly states that Eq. (1) follows ABXpy, and ABXpy is the original implementation used in ZeroSpeech 2015 and 2017, so treating it as the protocol of record is defensible. But it is a choice, and the paper does not cite an independent protocol specification. If ZeroSpeech 2021 intended a half-open interval, then the Libri-Light \"bug\" is not a bug and the fastabx default changes scores. The authors should acknowledge this contingency explicitly, or better, run the numerical comparison that would settle it.\n\nBottom line: this is a solid software paper with a reproducible artifact and a falsifiable bug claim. It deserves peer review and would benefit from a required conformance test. I'd cite it if I were doing ABX-based evaluation, and I'd probably bring it up in a reading group focused on speech representation evaluation methods. It is not a paradigm shift, but it is a workmanlike contribution that the community will actually use.","headline":"A genuinely useful ABX library with a sharp bug report, but the exact-replication claim needs numerical support before I'd trust the headline numbers.","tokens_in":10231,"tokens_out":1422,"would_cite":true,"duration_ms":17627,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"fastabx is a new Python library that computes ABX discriminability scores for arbitrary ON/BY/ACROSS tasks, exactly replicates the outputs of ABXpy, and ran the standard LibriSpeech phoneme ABX task about 65 times faster than ABXpy.","keywords":["fastabx","ABX discriminability","self-supervised speech representations","zero-resource evaluation","dynamic time warping","GPU parallelism","phoneme discrimination","representation learning"],"falsifier":"Run the same ABX task (e.g., LibriSpeech dev-clean, phoneme ON, speaker and context BY) through ABXpy and fastabx and compare per-cell error rates: any numerical difference, beyond floating-point rounding, refutes the exact-replication claim. Alternatively, recompute ZeroSpeech 2021 scores with features sliced to include the extra final frame; if scores are identical to published ones, the claimed Libri-Light bug has no observable effect.","tokens_in":9279,"feed_emoji":"⚡","tokens_out":6569,"duration_ms":59599,"temperature":0.7,"pith_summary":"This paper introduces fastabx, a Python library that computes ABX discriminability scores quickly and generically. ABX measures how cleanly a representation separates categories by asking whether samples from the same category are consistently closer to each other than to samples from another category, and it is a standard zero-resource way to evaluate self-supervised speech representations. The paper's central claims are that fastabx can build any ON/BY/ACROSS ABX task, that its outputs exactly replicate those of the older ABXpy implementation, and that it does so about 65 times faster on a standard speech benchmark. The paper also reports that the Libri-Light implementation used in past ZeroSpeech challenges sliced features one frame too short, and that fastabx can optionally reproduce that behavior.","feed_headline":"Faster ABX library replicates ABXpy and exposes Libri-Light bug","feed_subtitle":"A drop-in replacement turns hours-long ABX evaluations into minutes and flags an off-by-one error in past scores.","key_machinery":"The central object is the ABX task, parameterized by three conditions: ON (the attribute shared by A and X but not B), BY (shared by A, B, and X), and ACROSS (shared by A and B but not X). The machinery that carries the argument is a three-class pipeline: Dataset wraps samples and feature access; Task precomputes all valid triples per cell using Polars' lazy operations; Score computes d(x,a) < d(x,b) distances, with DTW alignment implemented as a PyTorch C++/CUDA extension using wavefront parallelism across diagonals. A second load-bearing piece is the frame-slicing convention I = {i in N | t_on <= t_i <= t_off} with t_i = Delta_t/2 + Delta_t i, which defines which feature frames belong to a triphone and is used to judge Libri-Light's slicing as one frame too short.","core_discovery":"The discovery is twofold. First, ABX evaluation can be made both general and fast: fastabx uses lazy dataframe operations to precompute task cells and a PyTorch C++/CUDA dynamic-time-warping kernel with wavefront parallelism, reducing the LibriSpeech dev-clean phoneme ABX task from 2 hours 12 minutes to 2 minutes 2 seconds while keeping ABXpy's interface and conventions. Second, the paper identifies an off-by-one error in the Libri-Light evaluation code: features were sliced with features[i_start:i_end] instead of features[i_start:i_end+1], so the last frame of every unit was dropped; the paper argues this follows from the same frame-index convention as ABXpy, and that it matters most for features with large time steps. fastabx replicates ABXpy exactly by default and offers an environment variable to reproduce Libri-Light's shorter slicing.","pith_inferences":["If the slicing bug is genuine, benchmark comparisons between models evaluated with Libri-Light may need rechecking, especially for representations with coarse temporal resolution (40 or 80 ms steps) where a dropped frame is a larger fraction of the unit.","A natural stress test is to recompute ZeroSpeech 2021 leaderboard scores with fastabx's default (ABXpy) slicing and compare ranking shifts; the paper does not provide this.","The same wavefront-parallel DTW approach could be adapted to other alignment-heavy evaluation metrics, such as word error rate or frame-level canonical correlation, where DTW is a bottleneck."],"forward_implications":["ABXpy users can switch to fastabx as a drop-in replacement, obtaining identical scores in a fraction of the time.","Past ZeroSpeech 2021 scores computed with Libri-Light were computed with one fewer frame per unit, so their absolute values may be slightly off; the paper does not quantify the effect on published numbers.","The generic ON/BY/ACROSS interface makes it straightforward to compute ABX scores in new domains or with new condition configurations, such as the context-invariant ABX of prior work that required rewriting Libri-Light.","The GPU DTW kernel makes large-scale ABX evaluations practical on a single machine, enabling layer-by-layer and contrast-by-contrast analyses."],"supporting_citations":[{"why":"Defines the ABX discriminability measure and the minimal-pair task.","marker":"Schatz et al., 2013"},{"why":"Formalizes ABX-discriminability measures and their applications.","marker":"Schatz, 2016"},{"why":"Introduces the Libri-Light implementation whose feature slicing is claimed to be one frame too short.","marker":"Kahn et al., 2020"},{"why":"Defines the ZeroSpeech 2021 benchmark that used Libri-Light for the phoneme ABX task.","marker":"Nguyen et al., 2020"},{"why":"Provides the LibriSpeech dev-clean dataset used for the speed comparison.","marker":"Panayotov et al., 2015"},{"why":"PyTorch is the runtime for the library's C++/CUDA extension.","marker":"Paszke et al., 2019"}],"fun_headline_variants":["Fast ABX library replicates ABXpy, exposes Libri-Light bug","fastabx: ABX tasks in minutes, not hours, and a bug found","New Python library speeds ABX by 65x, reveals slicing bug","ABX evaluation library: drop-in speedup plus off-by-one fix","fastabx: efficient ABX with a built-in bug detector"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's strongest claims depend on its assumption that ABXpy's frame-slicing rule, which includes the frame whose time is nearest to both the start and end timestamps, is the correct protocol; if the intended protocol was Libri-Light's shorter slice, then the exact-replication claim, the reported bug, and the speed comparison are all about different tasks.","fun_headline_variants_meta":{"raw":{"variants":["Fast ABX library replicates ABXpy, exposes Libri-Light bug","fastabx: ABX tasks in minutes, not hours, and a bug found","New Python library speeds ABX by 65x, reveals slicing bug","ABX evaluation library: drop-in speedup plus off-by-one fix","fastabx: efficient ABX with a built-in bug detector"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000244,"raw_usage":{"total_tokens":1494,"prompt_tokens":869,"completion_tokens":625,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":485,"completion_tokens_details":{"reasoning_tokens":526}},"tokens_in":485,"tokens_out":625,"duration_ms":5945,"temperature":1.0,"reasoning_tokens":526,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:43:35.317506+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same ABX task (e.g., LibriSpeech dev-clean, phoneme ON, speaker and context BY) through ABXpy and fastabx and compare per-cell error rates: any numerical difference, beyond floating-point rounding, refutes the exact-replication claim. Alternatively, recompute ZeroSpeech 2021 scores with features sliced to include the extra final frame; if scores are identical to published ones, the claimed Libri-Light bug has no observable effect.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the ABX discriminability measure and the minimal-pair task."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Formalizes ABX-discriminability measures and their applications."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"PyTorch is the runtime for the library's C++/CUDA extension."}],"review_version":1}