Pith. sign in

REVIEW 4 major objections 6 minor 12 references

This paper argues that a convex gated probe over all frozen layers makes probing nearly as informative as fine-tuning for audio self-supervised models, and that using this probe as a guide produces a transformer whose best semantic informat

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

CGP probing—layer-gating plus prototypes—closes much of the gap between frozen and fine-tuned audio SSL evaluation, and guides a re-engineered audio transformer (BAT) that improves on the authors' reproduced baselines.

T0 review reviewed 2026-08-02 challenge →

load-bearing objection A useful probing method and a careful baseline effort, undermined by an unsupported SOTA claim and by selection of all design choices on the same AS-20k benchmark used as a headline result. the 4 major comments →

arxiv 2602.16305 v2 pith:EOWBOHTS submitted 2026-02-18 cs.SD cs.LG

BAT: Better Audio Transformer Guided by Convex Gated Probing

Houtan Ghaffari , Lukas Rauch , Christoph Scholz , Paul Devos This is my paper
classification cs.SD cs.LG
keywords self-supervised learningaudio classificationprobingfrozen embeddingsAudioSetmasked latent regressionattention gatingtransformer
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish two linked results: that frozen-feature probing can be almost as informative as fine-tuning for audio SSL models, provided the probe reads every layer, and that once such a probe is trusted, a redesign of the audio SSL pipeline produces a transformer whose best information sits in later layers and transfers better. This matters because AudioSet-style results currently depend on fine-tuning, which the paper shows is hyperparameter-sensitive and hard to reproduce. If the probe is reliable, leaderboards could be built on cheap, standardized frozen evaluations instead. Guided by the probe, the paper's BAT model, with a new input frontend, sigmoid-gated attention, end-of-block targets, and a heavyweight ViT decoder, improves frozen and fine-tuned performance over reproduced baselines and narrows the probe-to-fine-tuning gap to a few points.

Core claim

The central discovery is that a convex gated probe can read a frozen audio transformer densely enough to nearly match fine-tuning, and that the same probe pinpoints which layers hold task-relevant information. Using the probe as a guide, the paper reworks the standard audio SSL pipeline, including input frontend, target construction, and decoder, and shows that a heavy ViT decoder plus a sigmoid gate in attention shifts the semantic peak from middle layers to late layers, making frozen embeddings markedly more linearly separable. The probe aggregates all layers with a learnable softmax-gating vector, compares patch and cls embeddings to learned prototypes, pools patch similarities via min an

What carries the argument

Convex Gated Probing (CGP) is the paper's main instrument: a learnable softmax-gating vector over layers aggregates L2-normalized embeddings from every block into one patch map and one cls embedding; cosine similarities to K prototypes are min-max pooled across tokens and concatenated with cls-prototype similarities, yielding 3K features for a linear classifier. The softmax weights expose which layers carry the task information, and the pooling lets spatially dispersed events contribute. Two architectural changes carry BAT: a sigmoid gate applied to attention-weighted values before the output projection, which removes attention sinks and enables end-of-block targets to outperform the previou

Load-bearing premise

The paper's conclusions rest on the assumption that the authors' reproductions of prior models' fine-tuning recipes and pretraining protocols are faithful enough to declare the published state-of-the-art numbers optimization artifacts rather than genuine representation quality.

What would settle it

Run a controlled comparison where two pretrained encoders are ranked by CGP, then fine-tuned across many hyperparameter settings; if the fine-tuning ranking ever reverses the CGP ranking with high confidence, CGP is not a faithful proxy. Alternatively, reproducing the previously reported SOTA fine-tuning results exactly, with the same code, recipe, and seeds, would collapse the claim that those gains were artifacts.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If CGP is a reliable proxy, audio SSL models can be evaluated without fine-tuning, removing hyperparameter choices and compute cost from leaderboard comparisons.
  • CGP's learned gating can diagnose underutilized layers, telling model developers when a lightweight decoder forces late encoder blocks to spend capacity on reconstruction.
  • A sufficiently expressive decoder shifts task-relevant information toward the final layer and makes frozen embeddings more linearly separable, which should improve transfer to downstream tasks.
  • Gated attention that removes attention sinks enables end-of-block SSL targets to beat the previous best MLP-only targets, showing that target construction and attention architecture are coupled.
  • Under a standardized reproduction of prior protocols, the new model consistently outperforms the reproduced baselines, and on at least one benchmark the frozen CGP result exceeds fine-tuning.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the ranking claim holds, AudioSet-style leaderboards built on fine-tuning may need to be re-examined; a frozen-probe ranking could become the default comparison, and SSL pretraining objectives could be optimized against CGP as a validation signal during training.
  • The decoder-capacity effect likely extends beyond audio: masked-image models with lightweight decoders may hide semantic information in middle layers, and a layer-wise gating probe could reveal it, suggesting a general design principle for masked pretraining.
  • A direct testable extension would train the same encoder with CNN versus ViT decoders at several capacities; the paper's hypothesis predicts CGP gating weights shift monotonically with decoder capacity and track fine-tuning rankings.
  • CGP's min/max pooling is tailored to multi-label audio events; for tasks where a single event fills the whole clip, alternative pooling over prototype similarities may need adaptation to preserve the method's advantage.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Convex Gated Probing (CGP), a frozen-feature evaluation method that aggregates all ViT layers through softmax weights, compares the resulting patch and cls embeddings with K learnable prototypes, and trains a linear classifier on min/max pooled similarities. It reports that CGP substantially outperforms linear probing and the prior Protobin method on EAT/SSLAM, and partially closes the gap to fine-tuning. Using CGP as a guide, the paper develops BAT, a modernized data2vec 2.0-style audio SSL model incorporating a new frontend, sigmoid gated attention, end-of-block targets, and a ViT decoder. The paper claims that BAT establishes new SOTA on audio benchmarks. The manuscript includes detailed hyperparameters and a standardized implementation, but all design ablations and K selection are conducted on AS-20k, which is also used as a headline benchmark.

Significance. If the empirical claims survive scrutiny, the contribution is significant: CGP would provide a cheaper and more transparent alternative to fine-tuning for evaluating audio SSL, and the gated-attention / decoder findings are actionable for future SSL design. The authors also make a useful reproducibility effort by replacing the legacy fairseq pipeline and by reporting standardized hyperparameters. However, the current evidence is not yet load-bearing: the AS-20k benchmark is used for both design selection and headline evaluation; the 'new SOTA' claim conflicts with the reported numbers in Table 5; and the baseline reproductions are acknowledged to be imperfect. The paper's value is therefore conditional on additional independent validation and a more careful framing of the claims.

major comments (4)
  1. [Abstract; §6, Table 5] The abstract's claim that BAT 'establish[es] new SOTA on audio benchmarks' is contradicted by the numbers in Table 5. Under the reported columns, BAT's fine-tuned AS-2M result (48.55) is below EAT (48.6) and SSLAM (50.2); on ESC-50 BAT (95.52) is below EAT (95.9) and SSLAM (96.2); and on SC-2 BAT (98.13) is below EAT (98.3). BAT exceeds the reported figures only on AS-20k (41.32 vs 40.9 for SSLAM). Section 6 itself concedes that 'BAT does not reach the peak reported figures.' The SOTA assertion should be removed or re-qualified, and the superiority claim re-framed as superiority under the authors' standardized reproduction protocol.
  2. [§5 (Tables 2–4), Fig. 2, §6 (Table 5)] Every BAT design choice and the CGP prototype count K are selected on AS-20k, and AS-20k is then used as a headline benchmark. §5 states 'All ablations are conducted on AS-20k utilizing CGP with 10k prototypes,' and Fig. 2 tunes K=10k on that same benchmark. Thus the AS-20k column of Table 5 is a selection artifact, not an independent measurement of CGP's reliability or of the BAT design choices. The ESC-50, SC-2, and AS-2M results are reported only for the already-selected model and cannot validate those choices. Please run ablations on a held-out split or on at least one dataset not used for selection, and separate validation from test when reporting the final table.
  3. [§4/§6, Tables 1 and 5] The comparison between BAT and EAT/SSLAM rests on the authors' reproductions, but the authors report that they could not reproduce the published SOTA fine-tuning numbers, and §6 states that they 'could not reproduce their pre-trained SSL model with their exact recipe.' If the baselines are not faithful to the original protocols, 'BAT consistently outperforms EAT and SSLAM' may reflect an unfavorable baseline rather than a genuine improvement. The manuscript should provide exact fine-tuning configurations, release reproduced checkpoints and code, and explicitly discuss how protocol changes (e.g., removing the 8e4 global-loss scaling) affect the comparison. The official reported numbers should remain visible as an upper reference in the table.
  4. [Tables 1–5] All results are single-run point estimates. Several key differences are small (Table 2: 35.03 vs 34.86; Table 4: 37.43 vs 37.52; Table 5 AS-2M FT: 48.55 vs 47.61/47.69). Given the paper's own emphasis on fine-tuning sensitivity and optimization artifacts, the absence of multiple seeds, error bars, or at least a variance analysis makes it difficult to establish that the claimed gains are statistically meaningful. Please report multiple runs for the main tables or provide confidence intervals for the central comparisons.
minor comments (6)
  1. [§2.1] The text says 'we propose CGB' in the probing subsection; this should be CGP. The same typo appears later in the paragraph ('CGB also resolves').
  2. [Table 5] The table header is hard to parse: 'FT CGP LP' is repeated four times without a clear grouping by dataset. Please add dataset subheadings and unify capitalization (AS-20K vs AS-20k).
  3. [Figure 2] The figure has no axis labels or numeric values, making the 'diminishing returns' and 'computation-performance trade-off' discussion difficult to verify. Please add axis labels and the actual mAP values.
  4. [Appendix A, Table 6] The table uses 'SPC-2' while the main text uses 'SC-2' or 'Speech Commands V2.' Please unify the notation.
  5. [References] The two data2vec references (2022a and 2022b) are listed with identical titles and appear to be the same conference paper; please disambiguate or merge.
  6. [§6] The reproducibility statement says 'We release the code upon acceptance or request during the review process.' For a paper whose main contribution is a reproducible evaluation protocol, an anonymous code/checkpoint link during review would materially help verification.

Circularity Check

1 steps flagged

AS-20k both selects BAT's design/K and is reported as the headline validation; that benchmark's evidence is circular, though other benchmarks remain independent.

specific steps
  1. fitted input called prediction [Section 5 Setup, Figure 2, and Table 5 (AS-20K columns)]
    "All ablations are conducted on AS-20k utilizing CGP with 10k prototypes. ... For Protobin and CGP, we use 10k prototypes. Figure 2 shows that this value provides a robust trade-off. ... Table 5: BAT consistently outperforms EAT and SSLAM baselines, showing substantial gains in the frozen embedding setting."

    Every design choice (frontend in Table 2, gated-attention/EOB targets in Table 3, decoder in Table 4) and the CGP prototype count K=10k are selected by maximizing CGP mAP on AS-20k. That same AS-20k column is then reported in Table 5 as evidence that BAT outperforms baselines. For the AS-20k row, BAT's CGP/LP scores are not independent predictions of the design; they are the selection criterion. Reporting them as validation that CGP-guided design works is therefore selection-on-the-test-set: the 'prediction' is forced by construction. The ESC-50, SC-2, and AS-2M results were not used in the ablations and are independent, so the circularity is partial rather than total.

full rationale

Walking the derivation chain: CGP itself is a supervised probe trained on labels, so its mAP is not a fitted parameter disguised as a prediction in the equation-level sense. The gated attention is adopted from external work (Qiu et al. 2025), not from a self-citation. The self-citations to Rauch et al. 2025a/b are contextual: the paper independently benchmarks CGP against Protobin and linear probing in Table 1, so those citations are not load-bearing. The one concrete circular step is the AS-20k selection loop: Section 5 states that all ablations use AS-20k with K=10k chosen from Figure 2, and Tables 2-4 choose the frontend, targets, and decoder by CGP mAP on that benchmark. Table 5 then presents AS-20k as part of the validation set, so BAT's AS-20k numbers reduce to the selection objective by construction. The ESC-50, SC-2, and AS-2M columns were not used in design selection and give the central claim independent support, which is why the score is 6 (partial circularity) rather than higher. The abstract's 'new SOTA' statement is contradicted by Table 5's reported EAT/SSLAM figures and by Section 6's admission that 'BAT does not reach the peak reported figures,' but that is an accuracy/consistency problem, not circularity.

Axiom & Free-Parameter Ledger

7 free parameters · 6 axioms · 0 invented entities

The central claims rest mainly on learned parameters and dataset-specific choices rather than on new physical or mathematical entities. CGP introduces no new external entity; its prototypes, layer weights, and K are fit to data. BAT's components are borrowed from prior work (gated attention, ViT decoder, frontend), and its configuration is selected on AS-20k. The most consequential non-standard assumption is that the authors' reproduction of EAT/SSLAM is faithful enough to attribute the reported SOTA gap to optimization artifacts.

free parameters (7)
  • CGP prototype count K = 10,000
    Chosen from Figure 2 AS-20k ablation; authors note the optimal count is dataset-dependent, so this hyperparameter is fit to the benchmark used in later claims.
  • CGP layer-gating weights a (softmax alpha) = learned per downstream task
    Fitted to labels; drives both CGP predictions and the layer-information diagnosis in Figure 4.
  • CGP prototype vectors P = K x D, learned
    Learned on labeled data; central to all CGP results and to the comparison against Protobin.
  • Global/local SSL loss weights = 1:1 (heuristic equal weighting)
    Authors deliberately remove D2V2's 8e4 global-loss factor and set equal weights; this is a hand-chosen training change affecting the model.
  • ViT decoder capacity = depth 6, heads 12, MLP ratio 4
    Selected from Table 4 ablation on AS-20k; yields the final BAT configuration.
  • Attention gate matrix W_G = learned
    Sigmoid gate inside MHSA, imported from Qiu et al.; trained end-to-end with no independent validation on audio.
  • Frontend configuration = TorchAudio mel + dB compression + local min-max
    Selected by Table 2 ablation; the paper claims improvement over the legacy pipeline and justifies it as dataset-independent.
axioms (6)
  • domain assumption Data2Vec 2.0 masked latent regression with EMA teacher is a sound SSL objective for audio
    The entire BAT pipeline assumes D2V2-style target construction; taken from prior literature, not re-derived here.
  • domain assumption CGP's softmax layer-gating plus prototype pooling is a faithful measure of frozen representation quality
    Core assumption of the paper; supported only by empirical correlations (Figure 4) and benchmark comparisons, not by theory.
  • ad hoc to paper Sigmoid gated attention (Qiu et al. 2025) transfers from LLMs to audio SSL and makes end-of-block outputs usable as SSL targets
    Imported hypothesis; Table 3 supports it only on AS-20k, and no audio-specific mechanism is validated.
  • domain assumption A heavy ViT decoder offloads reconstruction and shifts semantic information to later encoder layers
    Borrowed from MAE literature; Figure 4 evidence is derived from CGP's fitted layer weights, which is not independent evidence.
  • domain assumption AS-20k CGP mAP is a valid proxy for choosing architecture and for downstream transfer across benchmarks
    All design decisions and the K=10k choice use AS-20k; this assumes transfer to AS-2M, ESC-50, and SC-2.
  • domain assumption The reproduction of EAT/SSLAM with stated hyperparameters is faithful, so reported SOTA reflects optimization artifacts
    Load-bearing for the paper's evaluation narrative; the paper admits it could not reproduce even the pretrained SSL models with the exact recipes.

reviewed 2026-08-02 · how reviews work

0 comments
Cite this review

Pith. "Pith review of BAT: Better Audio Transformer Guided by Convex Gated Probing." pith.science (2026). https://pith.science/paper/EOWBOHTS

@misc{pith2026260216305,
  author       = {Pith},
  title        = {Pith review of: BAT: Better Audio Transformer Guided by Convex Gated Probing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EOWBOHTS}},
  note         = {Machine review of arXiv:2602.16305}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Probing is widely adopted in computer vision to faithfully evaluate self-supervised learning (SSL) embeddings, as finetuning may misrepresent their inherent quality. In contrast, audio SSL models still rely on finetuning because simple probing fails to unlock their full potential and alters their rankings when competing on AudioSet. Hence, a robust and efficient probing mechanism is required to guide the trajectory of audio SSL towards reliable and reproducible methods. We introduce Convex Gated Probing (CGP), a prototype-based method that significantly closes the gap between finetuning and probing in audio. CGP efficiently utilizes all frozen layers via a gating mechanism and exposes the location of latent task-relevant information. Guided by CGP as a reliable post-hoc evaluation probe, we rework the entire SSL pipeline of current best performing audio models that use legacy implementations of prior SSL methods. By refining data preprocessing, model architecture, and pretraining recipe, we introduce Better Audio Transformer (BAT), and establish new SOTA on audio benchmarks.

Figures

Figures reproduced from arXiv: 2602.16305 by Christoph Scholz, Houtan Ghaffari, Lukas Rauch, Paul Devos.

Figure 1
Figure 1. Figure 1: Convex Gated Probing (CGP). We illustrate the probing process of a spectrogram embedding for a ViT-Base backbone. CGP applies a learnable soft-gating vector (softmax) to compute a weighted sum of embeddings from all layers (L). The gating aggregates the hierarchy into a single representation, which is then compared against K prototypes. The cosine similarities of the patch embeddings are min-max pooled and… view at source ↗
Figure 2
Figure 2. Figure 2: CGP Ablation. Increasing the number of prototypes constantly improves results, but yields diminishing returns. The best computation-performance trade-off is dataset-dependent. 5. Ablations for a Better Audio Transformer The following ablation studies gradually introduce and eval￾uate the methodological enhancements in BAT. Setup. All pre-trainings are conducted exclusively on AS-2M, containing 1,932,574 au… view at source ↗
Figure 3
Figure 3. Figure 3: Impact of audio frontend. A recording containing the labels [Whimper, Gasp, Speech, Outside, urban or manmade]. (a) Our incorporated audio frontend: Mel-spectrogram with decibel compression and local min-max normalization, exhibiting clear spectral structure and high contrast. (b) Audio-MAE, EAT, and SSLAM: simple log, filtering, Mel-spectrogram, and global standardization. Note the artifacts and blurring,… view at source ↗
Figure 4
Figure 4. Figure 4: Layer-wise latent information. We display the layer-wise latent information quality across three models on AS-20k: (a) BAT with the lightweight CNN (best performer from [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Impact of gating on attentions. Gating distributes the attention better and focuses more on the token itself, rather than sinking into one token, primarily the cls-token. encoder should contribute more to masked token reconstruc￾tion rather than to learning high-level semantics (Huang et al., 2022), thereby diminishing the model’s effective ca￾pacity. This is particularly pronounced in regression-based MIM… view at source ↗
Figure 6
Figure 6. Figure 6: CGP weights for multi-class datasets. The weights are comparable to AS-20k weights we show in the main paper. We see that BAT shifts the information more towards the final layers. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

12 extracted references · 10 linked inside Pith

  1. [3]

    Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531,

    Hinton, G., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531,

    Pith/arXiv arXiv

  2. [4]

    P., Hunt, J

    Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y ., Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning.arXiv preprint arXiv:1509.02971,

    Pith/arXiv arXiv

  3. [10]

    Unmute the patch tokens: Rethinking probing in multi-label audio classification

    Rauch, L., Heinrich, R., Ghaffari, H., Miklautz, L., Moum- mad, I., Sick, B., and Scholz, C. Unmute the patch tokens: Rethinking probing in multi-label audio classification. arXiv:2509.24901, 2025a. Rauch, L., Heinrich, R., Moummad, I., Joly, A., Sick, B., and Scholz, C. Can masked autoencoders also listen to birds?Transactions on Machine Learning Researc...

    Pith/arXiv arXiv

  4. [11]

    R., Raj, B., Schuller, B

    Turian, J., Shier, J., Khan, H. R., Raj, B., Schuller, B. W., Steinmetz, C. J., Malloy, C., Tzanetakis, G., Velarde, G., McNally, K., Henry, M., Pinto, N., Noufi, C., Clough, C., Herremans, D., Fonseca, E., Engel, J., Salamon, J., Esling, P., Manocha, P., Watanabe, S., Jin, Z., and Bisk, Y . Hear: Holistic evaluation of audio representations. arXiv:2203.03022,

    Pith/arXiv arXiv

  5. [12]

    Speech Commands: A Dataset for Limited- V ocabulary Speech Recognition.arXiv:1804.03209,

    Warden, P. Speech Commands: A Dataset for Limited- V ocabulary Speech Recognition.arXiv:1804.03209,

    Pith/arXiv arXiv

  6. [2015]

    Beyond [CLS]: Exploring the true potential of masked image modeling representations

    Przewi˛ e´ zlikowski, M., Balestriero, R., Jasi´nski, W., ´Smieja, M., and Zieli ´nski, B. Beyond [CLS]: Exploring the true potential of masked image modeling representations. arXiv preprint arXiv:2412.03215,

    arXiv

  7. [2019]

    Beit v2: Masked image modeling with vector-quantized visual tokenizers.arXiv preprint arXiv:2208.06366,

    Peng, Z., Dong, L., Bao, H., Ye, Q., and Wei, F. Beit v2: Masked image modeling with vector-quantized visual tokenizers.arXiv preprint arXiv:2208.06366,

    Pith/arXiv arXiv

  8. [2021]

    Masked spectrogram modeling using masked autoencoders for learning general-purpose audio repre- sentation.arXiv:2204.12260,

    Niizumi, D., Takeuchi, D., Ohishi, Y ., Harada, N., and Kashino, K. Masked spectrogram modeling using masked autoencoders for learning general-purpose audio repre- sentation.arXiv:2204.12260,

    Pith/arXiv arXiv

  9. [2022]

    Vicreg: Variance- invariance-covariance regularization for self-supervised learning.arXiv preprint arXiv:2105.04906,

    Bardes, A., Ponce, J., and LeCun, Y . Vicreg: Variance- invariance-covariance regularization for self-supervised learning.arXiv preprint arXiv:2105.04906,

    Pith/arXiv arXiv

  10. [2023]

    W., Nanfack, G., and Belilovsky, E

    Hameed, H. W., Nanfack, G., and Belilovsky, E. Not Only the Last-Layer Features for Spurious Correlations: All Layer Deep Feature Reweighting.arXiv preprint arXiv:2409.14637,

    Pith/arXiv arXiv

  11. [2024]

    fairseq: A fast, extensible toolkit for sequence modeling.arXiv:1904.01038,

    Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., and Auli, M. fairseq: A fast, extensible toolkit for sequence modeling.arXiv:1904.01038,

    Pith/arXiv arXiv 1904

  12. [2025]

    Attention, please! revisiting attentive probing for masked image modeling.arXiv:2506.10178,

    Psomas, B., Christopoulos, D., Baltzi, E., Kakogeorgiou, I., Aravanis, T., Komodakis, N., Karantzalos, K., Avrithis, Y ., and Tolias, G. Attention, please! revisiting attentive probing for masked image modeling.arXiv:2506.10178,

    arXiv

This paper was first reviewed by deepseek-v4-flash on August 2, 2026.