{"id":"f257dbae-bb18-4603-9ff3-06f039aaaaad","arxiv_id":"2507.06794","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"HubertSoft embeddings encode phoneme identity and order at segment boundaries, detectable by a trained probe, with accuracy well above chance.","lead":"This paper tests whether HubertSoft, a self-supervised speech model, stores information about which phonemes appear at the start, middle, and end of each 20-millisecond audio window. A small neural network trained on Russian speech labels can predict phoneme order at segment boundaries, suggesting the embeddings carry temporal and coarticulatory detail.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim lacks an empirical baseline feature extractor, so the reported ordered accuracy may reflect trivial acoustic edge information in any 20 ms window rather than a HubertSoft-specific temporal encoding.","rationale":"The reader's weakest assumption was that manual phonetic segmentation is treated as exact ground truth despite acknowledged imprecision. That concern is real but conservative: random annotation noise would likely reduce the reported accuracies, especially center accuracy, rather than inflate the start/end ordered accuracy that the abstract emphasizes. The more load-bearing concern is the absence of any empirical baseline feature extractor. The paper compares ordered accuracy only against a hypothetical random-order model (Section 4), not against a real acoustic representation. If a simple spectral baseline matches the reported numbers, then the claim 'HubertSoft embeddings capture temporal order' is not a property of HubertSoft's learned representations but of any windowed audio signal, which would substantially weaken the paper's contribution. The reader's rationale did mention 'no empirical baseline feature extractor' among several weaknesses, but the weakest_assumption field focused on annotation noise; hence my agreement is partial. My proposed test would definitively settle whether the result is HubertSoft-specific. I do not propose changing the verdict because the reader's CONDITIONAL verdict already accommodates this concern, and a single baseline experiment would likely resolve it without requiring rejection.","tokens_in":6820,"tokens_out":17942,"duration_ms":215308,"concrete_test":"Re-run the identical probe architecture (four-layer feed-forward network, separate start/center/end heads, same loss and hyperparameter selection) and the same speaker-independent train/test split on standard acoustic features—specifically, 20 ms log-mel filterbank energies (e.g., 40 mel bands) computed from the same CORPRES audio windows, using the same triplet labels. Compare ordered, unordered, flexible-center, start, center, and end accuracies against Table 3. If the log-mel ordered accuracy is at least 0.53, the HubertSoft-specific temporal-order claim is not supported; if the baseline is substantially lower, the claim is strengthened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that HubertSoft boundary embeddings encode phoneme identity and temporal order (abstract, Section 5). The only baseline against which ordered accuracy (0.53) is judged is a simulated random-order baseline described in Section 4; no real non-HubertSoft feature extractor is evaluated. A 20 ms audio window by construction contains acoustic information from both its edges; a simple spectral feature (e.g., log-mel energies from the first and last 5 ms of the window) could plausibly predict the start and end labels and even some order information. If such a baseline reaches similar ordered accuracy, the result does not establish that temporal structure is specifically encoded in HubertSoft embeddings; it would only show that any windowed acoustic representation contains edge information. Since the paper's novelty rests on the claim that HubertSoft embeddings specifically preserve phoneme order, the current evidence cannot discriminate that hypothesis from the trivial alternative that raw acoustic features are sufficient. This is load-bearing because the conclusion over-attributes to HubertSoft without controlling for the acoustic content of the window.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper investigates whether HubertSoft embeddings encode temporal phonetic structure. Using the CORPRES Russian corpus, the authors extract non-overlapping 20 ms embeddings from frozen HubertSoft weights and label each window with a triplet of phonemes corresponding to the start, centre, and end of the window. A four-layer feed-forward network with three output heads is trained to predict these positions separately under a speaker-independent split. The authors report ordered accuracy of 0.5312, unordered accuracy of 0.9069, ordered flexible-centre accuracy of 0.7645, and per-position accuracies of 0.8823/0.6277/0.8563 for start/centre/end. They compare ordered accuracy with a simulated random-order baseline and interpret confusion patterns as evidence for articulatory and coarticulatory information in the embeddings.","tokens_in":7040,"tokens_out":7227,"duration_ms":80179,"significance":"If the central claim holds, the paper offers a useful probing methodology and contributes evidence about temporal information in SSL speech representations. The design has clear strengths: HubertSoft weights are frozen, the train/test split is speaker-independent, the corpus is publicly available and manually annotated, and the multiple metrics separately address content, order, and boundary localization. The confusion analyses are phonetically interpretable and connect the embedding behaviour to known articulatory and coarticulatory phenomena. The significance is currently limited because the result is not compared with any real non-HubertSoft feature extractor, and because the quantitative evidence lacks variance estimates, confidence intervals, and robustness checks against annotation noise.","major_comments":[{"comment":"The central claim that HubertSoft embeddings encode temporal order rests on a comparison of ordered accuracy (0.53) against a simulated baseline only; no real non-HubertSoft feature extractor is evaluated. Since every 20 ms window by construction contains acoustic information from its two edges, a simple spectral feature such as log-mel energies computed over the first and last few milliseconds could plausibly predict start and end labels and some order information. To support the abstract's claim that HubertSoft embeddings specifically capture temporal order, the authors should run the same probe on standard acoustic features (e.g., log-mel or MFCC) and, ideally, on HuBERT or Wav2Vec 2.0 embeddings. Without such a control, the reported ordered accuracy does not distinguish a HubertSoft-specific temporal encoding from generic acoustic edge information.","section":"Section 4, Table 3"},{"comment":"The manual segmentation noise acknowledged in Section 2 is not quantified, yet all metrics in Table 3 treat the manual boundaries as exact ground truth. Because start/centre/end labels are defined from boundary positions, a boundary shifted by one 20 ms frame changes the triplet label, so the reported ordered accuracy and the confusion patterns may partly reflect annotation behaviour rather than the embeddings' temporal content. This concern is made concrete by the example in Section 4.2, where the authors note that the boundary between i and l is off by one frame. Please report the number of test windows and speakers, per-speaker accuracy with confidence intervals, and a robustness experiment in which boundaries are perturbed by ±1 frame before re-labelling and re-evaluating.","section":"Section 2 and Section 4"},{"comment":"The simulated baseline is underspecified and the stated expected value is not reproducible from the text. For a typical border window containing two distinct phonemes with one repetition (e.g., a_p_p), a model that identifies the correct two-phoneme set with probability 0.9 per phoneme and then randomizes the multiset over the three positions would achieve roughly 0.27 expected ordered accuracy under a simple multiset-permutation model, not 0.22; other assumptions about false inclusions or position randomization give yet other values. The simulation procedure, the label distribution used, and the number of trials should be specified, and the baseline should be reported as a distribution rather than a single approximate point. If the corrected baseline is higher, the margin of the central comparison shrinks and should be discussed.","section":"Section 4, baseline calculation"}],"minor_comments":[{"comment":"The exact target phoneme list, the number of classes per position, and the criterion for selecting labels that are 'widely represented' are not reported; these details are needed to interpret the accuracy levels and to reproduce the experiment.","section":"Section 3"},{"comment":"It is unclear whether the reported results cover only border-spanning windows or all labelled windows, including those entirely inside a phoneme; since the paper focuses on boundaries, the composition and counts of the test set should be stated explicitly.","section":"Section 4, Table 3"},{"comment":"The application example reports 0.88 ordered accuracy on 17 frames, but the text does not state whether this recording comes from the held-out speakers or describe how frame-level predictions are aggregated into the reported accuracy; please clarify.","section":"Section 4.2"},{"comment":"The confusion matrices are difficult to read at the printed size, and symbols such as p' and a0 are not defined consistently in the captions or axes; please improve the figure legibility and label definitions.","section":"Figures 3 and 4"},{"comment":"Reference [9] (Deep Voice) does not appear to support the sentence on CTC-based phoneme boundary detection; consider replacing it with a more directly relevant CTC alignment or segmentation reference.","section":"Introduction, Reference [9]"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a conference-style contribution whose central idea is interesting, but the current evidence does not yet support the HubertSoft-specific temporal-encoding claim because of the missing empirical baseline and the absence of uncertainty quantification. Both concerns are fixable within the scope of the paper. I saw no evidence of duplicate publication or citation problems; the main request is for additional experiments and clearer reporting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a small, competent probing study. What is actually new is the triplet-label scheme: labelling 20 ms HubertSoft windows with the phonemes at start, centre, and end, then training a probe to predict all three positions. The flexible-centre metric is a sensible addition. The gap between unordered accuracy (0.91) and ordered accuracy (0.53) does suggest the embeddings retain some temporal order, and the confusion-pattern analysis is the best part of the paper. The explanations in terms of palatalization and coarticulation are careful and phonologically grounded, and the authors do acknowledge the manual segmentation uncertainty, which is more than many such papers do.\n\nThe soft spots are real but not fatal. The biggest one is the absence of any empirical baseline feature extractor. The simulated random-order baseline (0.22) only shows the probe beats a model that knows the phonemes but scrambles order. It does not rule out that a 20 ms log-mel window would yield similar ordered accuracy simply because the window edges sample the two adjacent sounds. That concern is legitimate and should be addressed with at least one simple acoustic baseline. Also, the manual segmentation is acknowledged but not quantified; a one-frame shift can change the start/centre/end labels, so the reported numbers may partly reflect annotation noise. There are no confidence intervals or significance tests, and the phoneme set is selected after the fact. These are fixable.\n\nI do not think the paper is circular or incoherent. The probe is a standard supervised readout, and the arbitrary 0.9 baseline is a heuristic, not a circular step. The stress-test worry is fair but slightly overstates the damage: even if raw audio also carries edge information, the paper's finding is about what the embeddings encode, and the confusion patterns go beyond trivial edge acoustics. Still, the missing baseline weakens the specificity of the claim.\n\nThis deserves a serious referee. It is an incremental but useful contribution to SSL probing, and the triplet-label methodology could be reused. I would send it to peer review with the expectation that the authors add a spectral baseline, error bars, and some quantification of segmentation noise. I would not cite it in my own work until those gaps are closed.","headline":"A small, honest probing study with a useful new triplet-label protocol, but the missing acoustic baseline and unquantified segmentation noise make the central claim thinner than the abstract suggests.","tokens_in":7520,"tokens_out":2067,"would_cite":false,"duration_ms":25032,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Frozen HubertSoft embeddings encode phoneme order within 20 ms windows, a probe shows, with ordered accuracy far above chance and strongest at segment boundaries.","keywords":["HubertSoft","temporal structure","phoneme boundaries","self-supervised speech embeddings","probing","CORPRES","coarticulation","Russian phonetics"],"falsifier":"Shuffle the order of the three phonemes in each training triplet (keeping the same phoneme set per window) and retrain the probe; if ordered accuracy stays close to 0.53, the temporal claim collapses because order contributes nothing over content. Alternatively, perturb the manual boundaries by ±20 ms and re-label the windows; if ordered accuracy drops toward the 0.22 random-order baseline, the result is an artifact of annotation noise rather than a property of the embeddings.","tokens_in":6622,"feed_emoji":"🗣️","tokens_out":5969,"duration_ms":55967,"temperature":0.7,"pith_summary":"The paper asks whether self-supervised speech embeddings do more than identify which phoneme is present: do they also record the order in which neighbouring phonemes occur in time? Using HubertSoft, a soft-clustering variant of HuBERT, and the manually annotated CORPRES Russian speech corpus, the authors label each 20 ms embedding window with a triplet giving the phoneme at its start, centre, and end. A small feed-forward network trained to predict all three positions reaches 0.53 ordered accuracy, well above the 0.22 expected from correct phoneme identities placed in random order, while unordered accuracy reaches 0.91. Start and end positions are decoded far better (0.88 and 0.86) than the centre (0.63), and the error patterns align with articulatory facts such as palatalization and vowel fronting. The paper concludes that boundary-frame HubertSoft embeddings encode both phoneme identity and temporal order.","feed_headline":"Probe decodes phoneme order from 20 ms speech embeddings","feed_subtitle":"Frozen HubertSoft windows map to start, centre and end phonemes at 53% ordered accuracy, far above chance.","key_machinery":"The central object is the triplet-labelled 20 ms window: each non-overlapping embedding window is tagged with the phoneme at its start, centre, and end according to the manual phonetic annotation. A four-layer feed-forward network with three output heads, one per position, is trained with summed cross-entropy, and the evaluation hinges on three metrics — ordered accuracy (all three positions correct in sequence), unordered accuracy (right phoneme set regardless of order), and ordered flexible-centre accuracy (correct start and end, centre allowed to match either). The ordered-versus-unordered contrast isolates temporal arrangement from phonetic content, and the start/end versus centre contrast localizes where in the window the temporal information lives.","core_discovery":"On the paper's own terms, the discovery is that frozen HubertSoft embeddings extracted from 20 ms windows at phoneme boundaries are temporally structured: a four-layer probe can decode the start, centre, and end phonemes of the window with an ordered accuracy of 0.53, compared with roughly 0.22 for a model that knows which phonemes are present but orders them randomly. The gap between unordered accuracy (0.91) and ordered accuracy (0.53) is the paper's evidence that the embedding space preserves sequence, not just a bag of phonetic features. Confusion patterns reinforce the claim: palatalized plosives are mistaken for their hard counterparts at the end of a closure, and a stressed /a/ after a palatalized consonant drifts toward [i]-like qualities at the window end, matching known Russian coarticulatory effects. This is presented as the first direct test of whether SSL embeddings encode the identity of the phoneme that begins or ends a segment.","pith_inferences":["The authors leave implicit that the triplet-probe diagnostic can rank any SSL speech model by temporal resolution; applying it to different layers of HuBERT or to wav2vec 2.0 would show whether ordered accuracy grows with layer depth or model family.","A natural testable extension is to replace the fixed 20 ms window with overlapping or variable-width windows, which could sharpen the centre-position accuracy and reveal how precisely boundaries are localized in embedding time.","If the temporal structure is genuinely coarticulatory, the same probe could be pointed at cross-linguistic corpora to ask whether embedding-level temporal encoding mirrors phonological typology — for instance, whether languages with richer consonant clusters show stronger boundary contrast.","The centre-position ambiguity (0.63) may indicate that the window's midpoint often falls inside a transition rather than a stable state; a soft label or distribution over centre phonemes might be a more honest target than a single hard label."],"forward_implications":["Ordered accuracy of 0.53 versus a 0.22 random-order baseline implies that the embedding space carries real sequential information, not just a content-based bag of phonemes.","Start and end positions decode at 0.88 and 0.86 while the centre lags at 0.63, so the temporal signal is concentrated at segment boundaries rather than in the window's interior.","The probe labels a novel 350 ms utterance with 0.88 ordered accuracy, showing that fine-grained phonetic labelling is achievable without retraining the SSL model.","Confusion patterns — p' vs p at segment end, /a/ fronting after palatalized consonants — indicate that the embeddings encode coarticulatory and articulatory detail, not static segments.","The method transfers across eight speakers with a four-speaker training split, so the temporal structure is speaker-general rather than memorized."],"supporting_citations":[{"why":"Introduces Wav2Vec 2.0, the SSL architecture family that motivates the question of what these embeddings encode.","marker":"[1]"},{"why":"Introduces HuBERT, the masked-prediction model whose soft-clustering variant supplies the embeddings under study.","marker":"[2]"},{"why":"Shows supervised probing of wav2vec 2.0 embeddings for phonetic features, the method this paper extends to temporal order.","marker":"[4]"},{"why":"Defines HubertSoft, the soft-clustering modification of HuBERT whose frozen embeddings are probed.","marker":"[12]"},{"why":"Supplies the CORPRES Russian speech corpus with its manual phonetic annotations that generate the triplet labels.","marker":"[13]"},{"why":"Provides the phonetic-transcription background stating that boundaries between acoustically similar sounds are placed approximately at the transition midpoint, the key caveat on ground truth.","marker":"[14]"}],"fun_headline_variants":["Phoneme order decoded from 20 ms HubertSoft windows","Temporal structure found in frozen speech embeddings","Boundary embeddings know start and end phonemes","Probe hits 53% ordered accuracy on phoneme triplets","Russian coarticulation effects visible in SSL embeddings"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the manual phoneme boundaries from CORPRES are accurate enough that a 20 ms window's start, centre, and end labels are true; the paper itself notes that boundaries between acoustically similar sounds are often placed approximately at the midpoint of the transition, so shifting a boundary by one frame changes the triplet and could inflate or distort the measured ordered accuracy.","fun_headline_variants_meta":{"raw":{"variants":["Phoneme order decoded from 20 ms HubertSoft windows","Temporal structure found in frozen speech embeddings","Boundary embeddings know start and end phonemes","Probe hits 53% ordered accuracy on phoneme triplets","Russian coarticulation effects visible in SSL embeddings"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000226,"raw_usage":{"total_tokens":1482,"prompt_tokens":970,"completion_tokens":512,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":586,"completion_tokens_details":{"reasoning_tokens":449}},"tokens_in":586,"tokens_out":512,"duration_ms":42915,"temperature":1.0,"reasoning_tokens":449,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:53:48.198699+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Shuffle the order of the three phonemes in each training triplet (keeping the same phoneme set per window) and retrain the probe; if ordered accuracy stays close to 0.53, the temporal claim collapses because order contributes nothing over content. Alternatively, perturb the manual boundaries by ±20 ms and re-label the windows; if ordered accuracy drops toward the 0.22 random-order baseline, the result is an artifact of annotation noise rather than a property of the embeddings.","supporting_citations":[{"cited_title":"Advances in Neural Information Processing Systems 33, 12449–12460 (2020) Hidden Temporal Structure of HubertSoft Embeddings 11","cited_arxiv_id":null,"evidence_quote":"Introduces Wav2Vec 2.0, the SSL architecture family that motivates the question of what these embeddings encode."},{"cited_title":"IEEE/ACM Trans","cited_arxiv_id":null,"evidence_quote":"Introduces HuBERT, the masked-prediction model whose soft-clustering variant supplies the embeddings under study."},{"cited_title":"In: Proc","cited_arxiv_id":null,"evidence_quote":"Shows supervised probing of wav2vec 2.0 embeddings for phonetic features, the method this paper extends to temporal order."},{"cited_title":"In: ICASSP 2022 – IEEE Int","cited_arxiv_id":null,"evidence_quote":"Defines HubertSoft, the soft-clustering modification of HuBERT whose frozen embeddings are probed."},{"cited_title":"In: Matouˇ sek, V., Mautner, P","cited_arxiv_id":null,"evidence_quote":"Supplies the CORPRES Russian speech corpus with its manual phonetic annotations that generate the triplet labels."},{"cited_title":"Phonetic transcription in theory and practice","cited_arxiv_id":null,"evidence_quote":"Provides the phonetic-transcription background stating that boundaries between acoustically similar sounds are placed approximately at the transition midpoint, the key caveat on ground truth."}],"review_version":1}