Pith. sign in

REVIEW 5 major objections 5 minor 21 references

Splitformer: An improved early-exit architecture for automatic speech recognition on edge devices

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Splitformer adds parallel downsampling branches to an early-exit ASR encoder and reports that they cut word error rate at every exit without adding inference time.

desk verdict Useful empirical idea, but the central causal claim is confounded by extra depth/parameters and the abstract overstates the timing; deserves review with major revision. read the letter →

arxiv 2506.18035 v1 pith:UALM2K4B submitted 2025-06-22 cs.CL cs.SDeess.AS

Maxence Lasbordes , Daniele Falavigna , Alessio Brutti This is my paper · ORCID
classification cs.CLcs.SDeess.AS
keywords early-exitautomaticspeechrecognitionconformerdownsamplingvariableframerateedgedevicesconnectionisttemporalclassificationworderror
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that adding a single parallel branch that processes a time-downsampled copy of its input, at the first and last exits of an early-exit speech recognizer, significantly lowers word error rate at every exit while leaving inference time essentially unchanged. The authors build a modified conformer encoder, Splitformer, that combines the output of each boundary block with a branch running at half the frame rate, mimicking the variable-frame-rate trick of the Zipformer without giving up the modularity that early exits need. On LibriSpeech, the worst case at the shallowest exit improves from 31.0% to 28.1% test-clean (51.0% to 48.3% test-other), and on TEDLIUM dev from 45.3% to 37.0%. The gains persist through all exits and are bought with only 5.2 million extra parameters (31.5M to 36.7M) and no extra decoding time at low exits. A sympathetic reading is that dynamic, resource-aware ASR on edge devices can get materially better accuracy at the same cost.

What carries the argument

The load-bearing mechanism is a parallel conformer branch that downsamples its input by a factor of two, processes it at the lower frame rate, upsamples back, and sums the result with the output of the main conformer block; this happens at the first and last encoder exits. The branch adds roughly 50% more parameters at the first exit (5.4M to 8.0M) but the total inference time does not grow, because the downsampled sequence costs proportionally less to process and CTC blank-token pruning dominates decoding time. It carries the argument by showing that the accuracy gain is not tied to running all layers at full temporal resolution.

What would settle it

Train a 14-layer early-exit conformer identical to Splitformer except with the parallel downsampling branches removed, keeping parameter count matched as closely as possible. If its layer-2 LibriSpeech test-clean WER is essentially the same as Splitformer's 28.1% (rather than near the 12-layer baseline's 31.0%), the proposed branches are not load-bearing; conversely, if removing them reverts WER toward 31.0% while depth is held fixed, the paper's mechanism is supported.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that early-exit encoders for ASR can keep the variable time-resolution processing of Zipformer-style architectures — where some layers operate on downsampled input at lower frame rates — by inserting parallel layers rather than altering the sequential backbone. The specific design that works best is two parallel branches, each a conformer block operating on a 2x downsampled input whose output is summed with the main block's output at the first and last exits of a 14-layer conformer encoder. Trained from scratch with a joint CTC loss over all exits, this Splitformer beats the 12-layer early-exit baseline at every exit on LibriSpeech and TEDLIUM, with the largest relative improvements at the shallowest exits, and its measured total decode times at the lowest exits are slightly below the baseline's. The authors attribute this to the parallel downsampling branch providing broader acoustic context without adding sequential depth.

Load-bearing premise

The paper's causal claim is that the parallel downsampling branches produce the accuracy gains, but all comparisons are against a 12-layer, 31.5M-parameter baseline, so if the extra two layers and 5.2M parameters (or some other confound) are what actually drive the improvement, the downsampling-specific story breaks.

Editorial extensions

If this is right

  • A device with budget for only two encoder layers would see LibriSpeech test-clean WER drop from 31.0% to 28.1%, and TEDLIUM dev from 45.3% to 37.0% — a large relative gain at the cost regime where edge devices operate.
  • The benefit is not confined to the exit with the parallel branch: WER improves at every exit (e.g., LibriSpeech test-clean at exit 12 goes from 5.1% to 4.8%), so no exit selection policy is penalized.
  • Measured total decode times at the first two exits are slightly lower for Splitformer than for the baseline, so the accuracy gain does not trade away the primary edge-device requirement.
  • The same downsampling idea also helps in a single-exit setting: the U-net-modified encoder without exits improves LibriSpeech test-clean from 6.1% to 4.4%, suggesting the mechanism is not specific to early exiting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A matched-capacity ablation (a 14-layer conformer without parallel branches, or a Splitformer pruned to 12 layers) is the direct way to test whether the mechanism or the extra depth is doing the work; the paper does not report one.
  • The mechanism suggests a cheap way to build confidence-aware exits: the parallel branch gives the shallowest layer a broader receptive field at almost no latency cost, which could in principle improve exit-policy reliability, but the paper only measures WER, not exit-selection behavior.
  • Because the gains appear at every exit, the approach may transfer to streaming or endpointed ASR settings, where early exits are more constrained; the paper does not test streaming.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes Splitformer, an early-exit ASR encoder built by adding parallel downsampling/upsampling branches to the first and last encoder blocks of a conformer-based early-exit baseline. The encoder is trained from scratch with a joint CTC loss over all exits, and evaluated on LibriSpeech and TEDLIUM against an early-exit conformer baseline and two pre-trained self-supervised models. The authors report WER improvements concentrated in the lowest exits, plus a per-exit analysis of execution time and FLOPs. The central empirical claim is that the parallel downsampling branches improve early-exit WER with only a small parameter increase and without affecting inference time.

Significance. If the attribution were cleanly established, the work would be a useful step toward resource-aware ASR architectures. The paper has clear strengths: it uses standard public benchmarks, provides code links for reproducibility, reports per-exit computational costs in a dedicated table, and compares against two established pre-trained models. However, the central causal claim is not isolated from increased model depth and capacity, and the inference-time claim is contradicted by the paper's own measurement table. Because the contribution is empirical, the main issues are addressable with additional controlled experiments rather than being structural flaws.

major comments (5)
  1. [§IV, Tables II–IV] The claim that introducing parallel downsampled branches is responsible for the WER improvements is not supported by the current comparison, because Splitformer has 14 conformer layers and 36.7M parameters whereas EE-baseline has 12 layers and 31.5M parameters. For example, at layer 2 on LibriSpeech test-clean the gain is 31.0→28.1, and at layer 2 on TEDLIUM dev it is 45.3→37.0; these gains could come entirely from the extra two conformer layers and the associated capacity. Please add a matched-capacity or matched-depth baseline (e.g., a 14-layer EE-baseline without parallel branches, or a parameter-matched ablation with the parallel branches removed) to isolate the effect of the downsampling branches.
  2. [§VI.A, Table V, Abstract] The statement that the approach improves WER 'without affecting the inference time' is contradicted by Table V: for LibriSpeech, layer-12 execution time increases from 1343 s to 1921 s (+43%) and layer-10 from 1414 s to 1898 s (+34%). Only the two lowest exits show comparable or slightly lower times (layer 2: 6112→5931 s; layer 4: 3217→3168 s). Please report the exit-dependent timing behavior explicitly and revise the abstract and conclusion to say that low exits are slightly faster while high exits are slower, or restrict the claim to the low-exit regime.
  3. [§VI, Tables III–IV] The word 'significantly' is used without statistical support. No repeated runs or error bars are reported, and some differences are small (e.g., LibriSpeech layer-12 test-clean WER 5.1→4.8; layer-8 test-other 16.6→16.3). To support the claimed significance, please report standard deviations over multiple seeds or use qualitative wording such as 'consistently lower' with a clear statement that no significance testing was performed.
  4. [§IV, Table I] The preliminary single-exit comparison between Conformer-baseline and the 'U-net modified' encoder changes the whole stack, not just the sampling structure: the U-net variant replaces the 12-layer stack with a block-based structure of different depth and parameter count. This comparison therefore cannot by itself show that variable frame-rate processing is the source of the gains. A matched-capacity single-exit ablation (e.g., a deeper plain conformer with the same parameter budget) would be needed to support the stated motivation.
  5. [§VII] The concluding sentence says that future work will evaluate the approach on early-exit models with 'much fewer parameters (e.g. with lower attention dimension, or reducing the size of the feed-forward network, or the number of attention heads).' This effectively concedes that the parameter regime relevant to edge devices has not been tested. Please add an explicit limitation statement in Section V or Section VII acknowledging that all reported models are comparatively large (31.5M–36.7M parameters) and that the results do not yet demonstrate applicability at the parameter scales typical of edge devices.
minor comments (5)
  1. [§VI.A] There is a typo: 'spliformer' should be 'Splitformer' in the sentence 'the spliformer ones are slightly lower at the lowest exits.'
  2. [§I] In the introduction, 'overcomes all previous confomer based models' should read 'conformer based models.'
  3. [Figure 1 caption] The caption contains a duplicated word: 'the best transcription is produced after by by only two encoder layers' should read 'after only two encoder layers.'
  4. [§V] The model name is written inconsistently as 'Wave2Vec2' in one place and 'Wav2Vec2' elsewhere; please unify the spelling.
  5. [§VII] The phrase 'we have proven' in the conclusions is too strong for an empirical result; 'we have shown' would be more appropriate.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the contribution is an empirical architecture comparison, and no prediction reduces to a fitted input or self-citation chain.

full rationale

The paper is an empirical architecture study, not a derivation. The central claim is that inserting parallel downsampling branches into an early-exit conformer encoder improves WER on LibriSpeech and TEDLIUM, and this is supported by direct comparisons in Tables III-V between EE-baseline and Splitformer. No parameter is fitted to the target benchmark and then renamed as a prediction. The authors cite their own prior work [1] for the joint early-exit training loss in Eq. (2) and for the resource-aware exit-selection setting, but the cited results are not used to argue that Splitformer is better; both compared architectures are trained with the same recipe, and the comparison is self-contained against standard external benchmarks. The selection of the best configuration after experimentation is model selection, not circularity. Potential confounds, such as the larger parameter count and depth of Splitformer compared with EE-baseline and the increased high-exit inference times in Table V, are correctness risks that an empirical paper should address, but they do not constitute a circular derivation. No quoted step in the paper reduces a predicted result to its own input by construction.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No analytic derivation is offered; the central result is empirical. The main unstated burden is that the performance gain is not causally isolated from added depth and parameters, and that benchmark WER is used as the optimization target for architectural choices.

free parameters (4)
  • number of parallel downsampling branches = 2
    Selected after 'experimentation with different configurations' (Section IV); the same benchmarks used for final evaluation informed the architecture choice.
  • downsampling factor for parallel branches = 2
    Chosen as the best compromise between complexity and performance; no held-out validation split is reported for this choice.
  • placement of parallel branches = first and last encoder exits
    The paper states that the best model includes two parallel downsampling layers in the first and last encoder outputs, with no ablation isolating this placement.
  • encoder depth and parameter count = 14 conformer layers, 36.7M parameters vs 12 layers, 31.5M for baseline
    Splitformer is not matched to EE-baseline in depth or capacity, so the WER gain cannot be fully attributed to the downsampling mechanism (Table II).
assumptions (5)
  • domain assumption The joint early-exit objective in Eq. (2), summing CTC losses from all exits, is a valid proxy for optimizing each exit's ASR quality.
    The paper's central comparison assumes that training from scratch with this loss is the right setup; this comes from the authors' prior work [1].
  • domain assumption LibriSpeech and TEDLIUM word error rates are sufficient indicators of edge-device ASR usefulness.
    These are the only evaluations; no latency or energy measurements on real edge hardware are reported.
  • domain assumption Removing training samples longer than 600 characters does not bias the evaluation.
    Data filtering is stated in Section V, but no analysis of its effect on results is provided.
  • ad hoc to paper The architecture selected as best in Section IV is not overfit to the evaluation benchmarks.
    The paper reports trying different configurations and choosing two parallel branches with downsampling factor 2 based on results, but no held-out validation split is reported for this architectural search.
  • standard math CTC loss, conformer blocks, and attention modules are used as in prior literature.
    The paper relies on conformer and CTC literature [7,16] without deriving those components.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Splitformer: An improved early-exit architecture for automatic speech recognition on edge devices." pith.science (2026). https://pith.science/paper/UALM2K4B

@misc{pith2026250618035,
  author       = {Pith},
  title        = {Pith review of: Splitformer: An improved early-exit architecture for automatic speech recognition on edge devices},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UALM2K4B}},
  note         = {Machine review of arXiv:2506.18035}
}
read the original abstract

The ability to dynamically adjust the computational load of neural models during inference in a resource aware manner is crucial for on-device processing scenarios, characterised by limited and time-varying computational resources. Early-exit architectures represent an elegant and effective solution, since they can process the input with a subset of their layers, exiting at intermediate branches (the upmost layers are hence removed from the model). From a different perspective, for automatic speech recognition applications there are memory-efficient neural architectures that apply variable frame rate analysis, through downsampling/upsampling operations in the middle layers, reducing the overall number of operations and improving significantly the performance on well established benchmarks. One example is the Zipformer. However, these architectures lack the modularity necessary to inject early-exit branches. With the aim of improving the performance in early-exit models, we propose introducing parallel layers in the architecture that process downsampled versions of their inputs. % in conjunction with standard processing layers. We show that in this way the speech recognition performance on standard benchmarks significantly improve, at the cost of a small increase in the overall number of model parameters but without affecting the inference time.

Figures

Figures reproduced from arXiv: 2506.18035 by the authors.

Figure 1
Figure 1. shows an example of an early-exiting network, where specific decoders process exits of intermediate layers. This work was partially funded by the PNRR project CN - HPC (Spoke 9) under the NRRP MUR program funded by the NextGenerationEU. Note that this architecture is suited to be implemented either over a distributed environment, consisting of models with a device specific number of layers (”resource aware”), or usi… view at source ↗
Figure 2
Figure 2. The U-net modified architecture of the audio encoder. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The Splitformer architecture of the audio encoder. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 19 canonical work pages

  1. [1]

    G. A. Wright, U. Cappellazzo, S. Zaiem, D. Raj, L. O. Yang, D. Falavigna, M. N. Ali, and A. Brutti, “Training early-exit architectures for 6https://pytorch.org/audio/main/tutorials/asr inference with cuda ctc decoder tutorial.html automatic speech recognition: Fine-tuning pre-trained models or training from scratch,” in Proc. of ICASSP Workshops , 2024, p...

    work page 2024

  2. [2]

    BranchyNet: Fast in- ference via early exiting from deep neural networks,

    T. Teerapittayanon, B. McDanel, , and H. Kung, “BranchyNet: Fast in- ference via early exiting from deep neural networks,” arXiv:1709.01686, 2017

    arXiv 2017

  3. [3]

    Distillation-based training for multi-exit architectures,

    M. Phuong and C. H. Lampert, “Distillation-based training for multi-exit architectures,” in Proc. of ICCV , 2019, pp. 1355–1364

    work page 2019

  4. [4]

    Fine-tuning Strategies for Faster Inference using Speech Self-Supervised Models: A Comparative Study

    S. Zaiem, R. Algayres, T. Parcollet, S. Essid, and M. Ravanelli, “Fine- tuning strategies for faster inference using self-supervised models: A comparative study,” arXiv:2303.06740, 2023

    work page Pith review arXiv 2023

  5. [5]

    Temporal early exiting for streaming speech commands recognition,

    R. Tang, K. V . S. M. Kumar, J. Xin, P. Vyas, W. Li, G. Yang, Y . Mao, C. Murray, and J. Lin, “Temporal early exiting for streaming speech commands recognition,” in Proc. of ICASSP , 2022

    work page 2022

  6. [6]

    Zipformer: A faster and better encoder for automatic speech recognition,

    Z. Yao, L. Guo, X. Yang, W. Kang, F. Kuang, Y . Yang, Z. Jin, L. Lin, and D. Povey, “Zipformer: A faster and better encoder for automatic speech recognition,” in Proc. of ICLR , 2024

    work page 2024

  7. [7]

    Conformer: Convolution-augmented transformer for speech recognition,

    A. Gulati, J. Qin, C. Chiu, N. Parmar, Y . Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y . Wu, and R. Pang, “Conformer: Convolution-augmented transformer for speech recognition,” in Proc. of Interspeech , 2020

    work page 2020

  8. [8]

    Branchformer: Parallel MLP-attention architectures to capture local and global context for speech recognition and understanding,

    Y . Peng, S. Dalmia, I. Lane, and S. Watanabe, “Branchformer: Parallel MLP-attention architectures to capture local and global context for speech recognition and understanding,” in Proc. of ICML , 2022

    work page 2022

Show all 21 references
  1. [9]

    Squeezeformer: An efficient transformer for automatic speech recognition,

    S. Kim, A. Gholami, A. Shaw, N. Lee, K. Mangalam, J. Malik, M. W. Mahoney, and K. Keutzer, “Squeezeformer: An efficient transformer for automatic speech recognition,” in Proc. of NeurIPS , 2022

    2022

  2. [10]

    ImageNet classification with deep convolutional neural networks,

    A. Krizhevsky, I. Sutskever, and G. Hinton, “ImageNet classification with deep convolutional neural networks,” Communications of the ACM , vol. 60, no. 6, pp. 84–90, 2017

    2017

  3. [11]

    HuBERT: self-supervised speech representation learning by masked prediction of hidden units,

    W. N. Hsu, B. Bolte, Y . Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “HuBERT: self-supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 29, pp. 3451–3460, 2021

    2021

  4. [12]

    Avoid overthinking in self- supervised models for speech recognition,

    D. Berrebbi, B. Yan, and S. Watanabe, “Avoid overthinking in self- supervised models for speech recognition,” in Proc. of ICASSP , 2022

    2022

  5. [13]

    WavLM: Large-scale self-supervised pre-training for full stack speech processing,

    S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, J. Wu, L. Zhou, S. Ren, Y . Qian, Y . Qian, M. Zeng, and F. Wei, “WavLM: Large-scale self-supervised pre-training for full stack speech processing,” IEEE Journal of Selected Topics in Si...

    2022

  6. [14]

    U-net: Convolutional networks for biomedical image segmentation,

    O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in Proc. of of International Confer- ence on medical image computing and computer-assisted intervention , 2015, pp. 234–241

    2015

  7. [15]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Proc. of NIPS, 2017, pp. 6000–6010

    2017

  8. [16]

    Con- nectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks,

    A. Graves, S. Fern ´andez, F. J. Gomez, and J. Schmidhuber, “Con- nectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks,” in Proc. of ICML , 2006

    2006

  9. [17]

    Librispeech: An ASR corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: An ASR corpus based on public domain audio books,” in Proc. of ICASSP , 2015, pp. 5206–5210

    2015

  10. [18]

    TED-LIUM 3: Twice as much data and corpus repartition for experi- ments on speaker adaptation,

    F. Hernandez, V . Nguyen, S. Ghannay, N. Tomashenko, and Y . Esteve, “TED-LIUM 3: Twice as much data and corpus repartition for experi- ments on speaker adaptation,” in SPECOM. Springer, 2018, pp. 198–208

    2018

  11. [19]

    Wav2vec 2.0: A framework for self-supervised learning of speech representations,

    A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “Wav2vec 2.0: A framework for self-supervised learning of speech representations,” in Proc. of NeurIPS , 2020

    2020

  12. [20]

    Neural machine translation of rare words with subword units,

    S. R., B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” in Proc. of ACL , 2016, pp. 1715––1725

    2016

  13. [21]

    Adam: A method for stochastic optimization,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv:1412.6980, 2014

    2014 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.