Pith. sign in

REVIEW 4 major objections 5 minor 73 references

Re-Parameterization of Lightweight Transformer for On-Device Speech Emotion Recognition

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that training a lightweight Transformer with an extra linear layer inserted before its feed-forward network, then exactly merging that layer away at inference, improves speech-emotion-recognition accuracy without…

desk verdict Algebra is exact and the trick is worth trying, but 'consistently improves' is unsupported by the reported tables because of post hoc selection and missing variance. read the letter →

arxiv 2411.09339 v1 pith:U3CE7AZN submitted 2024-11-14 cs.SD cs.CLeess.AS

Zixing Zhang , Zhongren Dong , Weixiang Xu , Jing Han This is my paper · ORCID
classification cs.SDcs.CLeess.AS
keywords speechemotionrecognitionTransformerre-parameterizationhigh-rankfactorizationlightweightmodelcompressionon-devicedeploymentIEMOCAP
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Speech-emotion recognition models small enough for phones and IoT devices normally lose accuracy when compressed. This paper tries to recover that accuracy without enlarging the deployed model: during training, a linear layer is inserted before the feed-forward network (and optionally other modules) to expand parameter count, and during inference that layer is exactly multiplied into existing weights and removed. The authors call the expansion High-Rank Factorization (HRF) and the exact reverse deHRF, and report that across ConvTransformer, Conformer, and SpeechFormer on the IEMOCAP, M3ED, and DAIC-WOZ datasets, HRF-trained lightweight models beat plain lightweight models in most configurations and can approach or exceed much larger original models. If true, this gives edge-device speech emotion recognition a training procedure that lifts accuracy at no added deployment cost.

What carries the argument

The machinery is High-Rank Factorization (HRF) and its exact inverse, de-High-Rank Factorization (deHRF), built on the identity that stacked affine layers with no nonlinearity collapse into one affine layer: W = W2W1, b = W2b1 + b2. The inserted layer is high-rank because it expands the intermediate width r to 2n, 4n, or 8n, the opposite of low-rank factorization's shrinking bottleneck. The paper places this expansion before the first or second fully connected layer of the FFN, or before QKV, projection, and classification-head layers, and fuses it away at inference. The inserted layer therefore changes only the training trajectory and vanishes at inference.

What would settle it

Train the same lightweight model with and without HRF under matched optimizer, learning-rate schedule, epochs, and seeds, then merge the HRF weights and evaluate on held-out test sets; the central claim fails if the merged model does not beat the directly trained model, or if its outputs differ from the training-phase model beyond floating-point roundoff.

Watch

Extended reading notes

Core claim

The central claim is that a purely linear re-parameterization can make a tiny Transformer generalize better after the inserted parameters are removed. For a fully connected layer, the paper replaces the single affine map by two or three affine maps with an expanded intermediate width (r = 2, 4, or 8 times the output width) and no activation between them; because stacked linear maps compose to a single linear map, deHRF exactly recovers the original weight and bias. The resulting inference model is structurally identical to the lightweight baseline, with no extra parameters or FLOPs. The reported gain is empirical: in most tested settings the merged model outperforms the same tiny model trained directly, and the best FFN-plugged configurations land close to or above models with roughly a hundred times more parameters on the same datasets.

Load-bearing premise

The load-bearing premise is empirical: inserting a linear layer with no activation changes the training dynamics so that the exact merged model generalizes better than the same tiny architecture trained directly, and that benefit survives across datasets and model variants.

Editorial extensions

If this is right

  • After merging, the deployed model has exactly the same parameter count, FLOPs, and architecture as the plain lightweight model, so on-device deployment adds no memory or compute overhead.
  • HRF-plugged FFN layers improve accuracy over plain tiny models in most settings across three datasets and three Transformer variants, with best configurations approaching or exceeding the original larger models.
  • Placing the extra linear layer on the second FFN layer (FFN2) works better than the first FFN layer in most cases, while placing it on the attention QKV module gives the least benefit.
  • Using more than one inserted linear layer does not help; one extra linear layer gives the best results across models and datasets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the merge is exact for any chain of linear layers, the same training-time expansion should transfer to other linear projections in a Transformer, such as token or value projections; this is directly testable with the paper's recipes.
  • The paper interprets its expansion-ratio trend as double descent, but a matched-budget comparison against plain training with identical optimizer, epochs, and seeds would separate a genuine generalization gain from an implicit change in effective training effort.
  • Nothing in the method prevents combining it with pruning or distillation after training, since the deployed weights are just the ordinary compact weights; stacking these procedures is a natural follow-up the paper does not report.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a training-time re-parameterization scheme for lightweight Transformers in speech emotion recognition. The High-Rank Factorization (HRF) process inserts an extra linear layer before the following fully connected layer of selected modules (FFN, QKV, Projection, CLS, etc.), and the deHigh-Rank Factorization (deHRF) process exactly merges that insertion back into the adjacent layer at inference, recovering the original architecture with no added parameters or FLOPs. The method is evaluated on ConvTransformer, Conformer, and SpeechFormer on IEMOCAP, M3ED, and DAIC-WOZ. The algebraic merge in Eq. (6) and Algorithm 2 is correct, and the reported parameter/FLOP counts confirm that inference cost is unchanged. The paper's main empirical claim is that HRF 'consistently improves' lightweight Transformers and can make them comparable to much larger models.

Significance. If established with adequate statistical support, the proposal would be practically useful for on-device SER: it is a simple, drop-in training-time modification with zero inference overhead, and the exact-merge construction is transparent and reproducible from the pseudocode. The paper also makes a useful comparison across three Transformer families and three datasets. However, the central claim is currently stronger than the evidence: the reported tables show multiple configurations underperforming the lightweight baseline, and no variance information or significance testing is provided. The contribution is therefore better characterized as 'some HRF placements may improve some lightweight Transformers' rather than the consistent improvement claimed in the abstract. The theoretical motivation in Section III-C is not sufficient to close this gap, because the cited results concern optimization acceleration and scaling at large model sizes, not generalization for 3K–10K parameter models.

major comments (4)
  1. [Abstract and Section IV-D, Tables III–V] The claim that HRF 'consistently improves' lightweight Transformers is contradicted by the paper's own results. For example, SpeechFormer with FFN1&FFN2 on IEMOCAP gives WF1 0.503 versus 0.528 for the lightweight baseline, and SpeechFormer with CLS on IEMOCAP gives 0.527 versus 0.528. On M3ED, Conformer with CLS and with ALL gives 0.367 versus 0.368, and SpeechFormer with FFN1&FFN2 gives 0.354 versus 0.358. On DAIC-WOZ, ConvTransformer with ALL gives 0.530, equal to the lightweight baseline. The text in Section IV-D itself uses 'in most cases,' which is a more accurate statement. The abstract and conclusion should be reworded, or the claim needs a formally specified rule that shows consistent improvement.
  2. [Section IV-A and Tables III–V] The paper states that five random seeds were used and results averaged, but all reported numbers are point estimates with no standard deviations, confidence intervals, or significance tests. Many of the reported differences are extremely small (e.g., 0.367 vs. 0.368 on M3ED), and such differences are indistinguishable from seed-level noise. The authors should report variance across seeds, provide confidence intervals, or perform paired significance tests. Without this, the empirical premise of the paper is not established.
  3. [Section III-B and Tables III–V] The headline improvements appear to be selected post hoc as the best result among many HRF module/placement variants (FFN1, FFN2, FFN1&FFN2, QKV, Project, CLS, ALL, and for Conformer FFN-M variants). Reporting the maximum over roughly 8–11 configurations per model-dataset introduces selection bias, and no multiple-comparison control or held-out model-selection procedure is described. To support the central claim, the authors should either pre-specify a fixed HRF placement, validate the chosen placement on a separate split, or otherwise account for the number of configurations explored.
  4. [Section III-C] The theoretical motivation does not support the claimed generalization benefit. The Arora et al. result concerns optimization-speed acceleration from overparameterized linear networks, not improved final generalization. The scaling-law citations concern how growing parameter count, data, and compute affect large-model performance, which is not the setting of a 3K–10K parameter model with a single inserted linear layer. This material should be presented as heuristic motivation rather than a theoretical guarantee, and the case for generalization must come from the experiments. As it stands, the section does not mitigate the weakness of the empirical evidence.
minor comments (5)
  1. [Section III-D, Eq. (3)] The dimension notation in Eq. (3) is inconsistent: for a layer with m input neurons and n output neurons, the weight matrix should be in R^{n×m}, not R^{m×n}; the same issue affects the description of r in the following paragraph.
  2. [Section III-D and Algorithm 1] The variable r is described as the expansion scale and later as 'r = 2n, 4n, or 8n', but Algorithm 1 uses Linear(m, r×n), where r is a multiplicative factor. Please clarify whether r denotes the intermediate dimension or the multiplier, and align the notation throughout.
  3. [Section IV-E] The phrase 'extend one HRF layer into two and three layers' uses a different notion of layer count than Section III-D, where N=3 refers to three factor matrices within one HRF expansion. This overloading of N is confusing and should be disambiguated.
  4. [Figure captions] The captions of Figures 4–6 contain stray numeric strings ('5.2, 12.2 13.1, 12.2 20.1, 12.2') that appear to be unintended artifacts; these should be removed or explained.
  5. [Throughout] There are several typos and formatting errors, including 'Transfering' in the Figure 1 caption, 'SpeechF ormer' in the Table V title, 'HRF FNN' for 'HRF FFN' in Section IV-D, and 'matrics' for 'matrices' in Section IV-D. A careful proofread is needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the deHRF merge is exact algebra and the performance gains are empirical observations, not derived from fitted inputs.

full rationale

The paper's load-bearing inference-time claim is the algebraic equivalence in Section III-E: Y* = sigma(W2(W1X+b1)+b2) = sigma((W2W1)X+(W2b1+b2)), so Y = Y*. This is an exact equality by construction, but it only establishes that the merged single-layer model computes the same function as the two-layer training model; it is not used to derive the reported accuracy gain. The claimed benefit of inserting an activation-free linear HRF layer is presented as an empirical finding (Tables III-V and the ablation figures), with no parameter fitted to the reported metrics and then renamed as a prediction. The theoretical discussion in Section III-C cites Arora et al. and scaling-law papers as motivation; those citations are external and are not used to prove that this particular insertion must improve generalization, so no load-bearing self-citation or imported uniqueness theorem is present. The authors' own prior work is cited only for non-central items, such as [66] for the acoustic feature configuration, and nothing in the central claim reduces to those citations. Weaknesses such as post hoc selection of the best HRF module and the absence of error bars are concerns about statistical evidence and selective reporting, not circular reasoning. Therefore the derivation chain is self-contained in the relevant sense: the only formal result (the merge identity) is independent of the empirical claim, and the empirical claim is not obtained by construction from its own inputs.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The method itself introduces no new fitted constants or physical entities; the expansion ratio, number of HRF layers, and module placement are hyperparameters selected through ablations on the same datasets. The theoretical motivation imports standard linear-overparameterization and scaling-law assumptions from cited work. The key design axiom is that no nonlinearity is inserted between HRF and the following layer, which makes the reconstruction exact but also limits what the extra parameters can express.

free parameters (3)
  • HRF expansion ratio r = 8 (default); 2 and 4 tested
    Sets the width of the inserted linear layer to r times the output dimension; chosen as a hyperparameter and analyzed in Section IV-E, Figures 4-6.
  • Number of HRF layers N = 1 (best); 2 and 3 tested
    Depth of the inserted linear stack; Section IV-E, Figures 7-9, reports one layer as best.
  • HRF module placement = best per model and dataset (e.g., FFN2, FFN-M1, Project)
    Which modules receive HRF is selected from the full results in Tables III-V after seeing test-set performance; Section IV-E calls this the 'best experimental setting'.
assumptions (5)
  • domain assumption Linear overparameterization improves optimization or generalization of compact networks.
    Invoked in Section III-C via Arora et al. and scaling-law references; not proven for the specific 3K-10K parameter Transformers used here.
  • ad hoc to paper HRF layers are inserted without intermediate activations, making the merge exact.
    The no-activation design is chosen to allow lossless reconstruction; it is not required by the Transformer architecture.
  • standard math Matrix multiplication of two linear layers is associative and exact.
    Used in Eq. (6) and Algorithm 2; this is a standard algebraic identity.
  • domain assumption The three Transformer baselines and 78-dimensional LMFB features are reliable representatives for SER.
    Used to justify the experimental design in Section IV; comparisons with prior SOTA are intended to validate the baselines.
  • domain assumption DAIC-WOZ can be evaluated as a clip-level binary classification task on the validation set.
    Section IV-A describes segmenting patient voices into 10 s clips and reporting validation MF1, which deviates from standard utterance-level evaluation and may affect comparability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Re-Parameterization of Lightweight Transformer for On-Device Speech Emotion Recognition." pith.science (2026). https://pith.science/paper/U3CE7AZN

@misc{pith2026241109339,
  author       = {Pith},
  title        = {Pith review of: Re-Parameterization of Lightweight Transformer for On-Device Speech Emotion Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U3CE7AZN}},
  note         = {Machine review of arXiv:2411.09339}
}
read the original abstract

With the increasing implementation of machine learning models on edge or Internet-of-Things (IoT) devices, deploying advanced models on resource-constrained IoT devices remains challenging. Transformer models, a currently dominant neural architecture, have achieved great success in broad domains but their complexity hinders its deployment on IoT devices with limited computation capability and storage size. Although many model compression approaches have been explored, they often suffer from notorious performance degradation. To address this issue, we introduce a new method, namely Transformer Re-parameterization, to boost the performance of lightweight Transformer models. It consists of two processes: the High-Rank Factorization (HRF) process in the training stage and the deHigh-Rank Factorization (deHRF) process in the inference stage. In the former process, we insert an additional linear layer before the Feed-Forward Network (FFN) of the lightweight Transformer. It is supposed that the inserted HRF layers can enhance the model learning capability. In the later process, the auxiliary HRF layer will be merged together with the following FFN layer into one linear layer and thus recover the original structure of the lightweight model. To examine the effectiveness of the proposed method, we evaluate it on three widely used Transformer variants, i.e., ConvTransformer, Conformer, and SpeechFormer networks, in the application of speech emotion recognition on the IEMOCAP, M3ED and DAIC-WOZ datasets. Experimental results show that our proposed method consistently improves the performance of lightweight Transformers, even making them comparable to large models. The proposed re-parameterization approach enables advanced Transformer models to be deployed on resource-constrained IoT devices.

Figures

Figures reproduced from arXiv: 2411.09339 by the authors.

Figure 3
Figure 3. Detailed illustration of the re-parameterization process [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 12
Figure 12. It was observed that under all scenarios the HRF [PITH_FULL_IMAGE:figures/full_fig_p009_12.png] view at source ↗
Figure 4
Figure 4. Results of WF1 when applying different expansion ratios of HRF to different modules of ConvTransformer (a), Conformer (b), or SpeechFromer (c) on the IEMOCAP dataset. (a) ConvTransformer (b) Conformer (c) SpeechFormer 5.2, 12.2 13.1, 12.2 20.1, 12.2 [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figures from the paper (8 more)
Figure 5
Figure 5. Figure 5: Results of WF1 when applying different expansion ratios of HRF to different modules of ConvTransformer (a), Conformer (b), or SpeechFromer (c) on the M3ED dataset. (a) ConvTransformer (b) Conformer (c) SpeechFormer 5.2, 12.2 13.1, 12.2 20.1, 12.2 [PITH_FULL_IMAGE:figu…
Figure 6
Figure 6. Figure 6: Results of MF1 when applying different expansion ratios of HRF to different modules of ConvTransformer (a), Conformer (b), or SpeechFromer (c) on the DAIC-WOZ dataset. REFERENCES [1] V. Sharma, T. G. Tan, S. Singh, and P. K. Sharma, “Optimal and privacy-aware resource …
Figure 7
Figure 7. Figure 7: Results when applying different numbers of HRF layer to the second feedforward layers of ConvTransformer (a), Conformer (b), or SpeechFromer (c) on the IEMOCAP dataset. (a) ConvTransformer (b) Conformer (c) SpeechFormer [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Results when applying different numbers of HRF layer to the second feedforward layers of ConvTransformer (a), Conformer (b), or SpeechFromer (c) on the M3ED dataset. (a) ConvTransformer (b) Conformer (c) SpeechFormer [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: Results when applying different numbers of HRF layer to the second feedforward layers of ConvTransformer (a), Conformer (b), or SpeechFromer (c) on the DAIC-WOZ dataset [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 10
Figure 10. Figure 10: Performance of re-parameterized lightweight ConvTransformer (a), Conformer (b), or SpeechFromer (c) with diverse [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]
Figure 11
Figure 11. Figure 11: Performance of re-parameterized lightweight ConvTransformer (a), Conformer (b), or SpeechFromer (c) with diverse [PITH_FULL_IMAGE:figures/full_fig_p012_11.png]
Figure 12
Figure 12. Figure 12: Performance of re-parameterized lightweight ConvTransformer (a), Conformer (b), or SpeechFromer (c) with diverse [PITH_FULL_IMAGE:figures/full_fig_p012_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

73 extracted references · 62 canonical work pages

  1. [1]

    Optimal and privacy-aware resource management in artificial intelligence of things using osmotic computing,

    V . Sharma, T. G. Tan, S. Singh, and P. K. Sharma, “Optimal and privacy-aware resource management in artificial intelligence of things using osmotic computing,” IEEE Transactions on Industrial Informatics , vol. 18, no. 5, pp. 3377–3386, May 2022. 11 (a) ConvTransformer (b) Conformer (c) SpeechFormer Fig. 7: Results when applying different numbers of HRF ...

    work page 2022

  2. [2]

    Empowering things with intelligence: A survey of the progress, challenges, and opportunities in artificial intelligence of things,

    J. Zhang and D. Tao, “Empowering things with intelligence: A survey of the progress, challenges, and opportunities in artificial intelligence of things,” IEEE Internet of Things Journal , vol. 8, no. 10, pp. 7789–7817, May 2021

    work page 2021

  3. [3]

    Enable deep learning on mobile devices: Methods, systems, and applications,

    H. Cai, J. Lin, Y . Lin, Z. Liu, H. Tang, H. Wang, L. Zhu, and S. Han, “Enable deep learning on mobile devices: Methods, systems, and applications,” ACM Transactions on Design Automation of Electronic Systems, vol. 27, no. 3, pp. 20:1–20:50, Mar. 2022

    work page 2022

  4. [4]

    Tinyad: Memory- efficient anomaly detection for time series data in industrial iot,

    Y . Sun, T. Chen, Q. V . H. Nguyen, and H. Yin, “Tinyad: Memory- efficient anomaly detection for time series data in industrial iot,” IEEE Transactions on Industrial Informatics , pp. 1–12, May 2023

    work page 2023

  5. [5]

    Real-time speech emotion analysis for smart home assistants,

    R. Chatterjee, S. Mazumdar, R. S. Sherratt, R. Halder, T. Maitra, and D. Giri, “Real-time speech emotion analysis for smart home assistants,” IEEE Transactions on Consumer Electronics , vol. 67, no. 1, pp. 68–76, Feb. 2021

    work page 2021

  6. [6]

    Speech emotion detection using iot based deep learning for health care,

    Z. Tariq, S. K. Shah, and Y . Lee, “Speech emotion detection using iot based deep learning for health care,” in Proc. 6th Conference on Big Data (Big Data) . Los Angeles, CA, USA: IEEE, Dec. 2019, pp. 4191– 4196

    work page 2019

  7. [7]

    Iot-enabled WBAN and machine learning for speech emotion recognition in patients,

    D. D. Olatinwo, A. M. Abu-Mahfouz, G. P. Hancke, and H. C. Myburgh, “Iot-enabled WBAN and machine learning for speech emotion recognition in patients,” Sensors, vol. 23, no. 6, p. 2948, Mar. 2023

    work page 2023

  8. [8]

    Emotion-aware connected health- care big data towards 5g,

    M. S. Hossain and G. Muhammad, “Emotion-aware connected health- care big data towards 5g,” IEEE Internet of Things Journal , vol. 5, no. 4, pp. 2399–2406, Aug. 2018

    work page 2018

Show all 73 references
  1. [9]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Proc. 31th Annual Conference on Neural Information Processing Systems (NeuIPS) . Long Beach, CA, USA: MIT Press, Dec. 2017, pp. 5998–6008

    2017

  2. [10]

    Pre-trained models for natural language processing: A survey,

    X. Qiu, T. Sun, Y . Xu, Y . Shao, N. Dai, and X. Huang, “Pre-trained models for natural language processing: A survey,” Science China Technological Sciences, vol. 63, no. 10, pp. 1872–1897, Sep. 2020

    2020

  3. [11]

    Transformers in vision: A survey,

    S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah, “Transformers in vision: A survey,” ACM Computing Surveys , vol. 54, no. 10s, pp. 1–41, Jan. 2022

    2022

  4. [12]

    Dawn of the transformer era in speech emotion recognition: Closing the valence gap,

    J. Wagner, A. Triantafyllopoulos, H. Wierstorf, M. Schmitt, F. Burkhardt, F. Eyben, and B. W. Schuller, “Dawn of the transformer era in speech emotion recognition: Closing the valence gap,” IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 45, no. 9, pp. 10...

    2023

  5. [13]

    A survey of large language models,

    W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y . Hou, Y . Min, B. Zhang, J. Zhang, Z. Dong et al., “A survey of large language models,” arXiv:2303.18223, 2023

    2023 arXiv

  6. [14]

    Language models are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askellet al., “Language models are few-shot learners,” in Proc. 34th Annual Conference on Neural Information Processing Systems (NeuIPS) . Virtual: MIT Press, Dec. 2020,...

    2020

  7. [15]

    Pangu- σ: Towards tril- lion parameter language model with sparse heterogeneous computing,

    X. Ren, P. Zhou, X. Meng, X. Huang, Y . Wang, W. Wang, P. Li, X. Zhang, A. Podolskiy, G. Arshinov et al. , “Pangu- σ: Towards tril- lion parameter language model with sparse heterogeneous computing,” arXiv:2303.10845, 2023

    2023 arXiv

  8. [16]

    Training compute-optimal large language models,

    J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark et al., “Training compute-optimal large language models,” arXiv:2203.15556, 2022

    2022 arXiv

  9. [17]

    Model compression and hardware acceleration for neural networks: A comprehensive survey,

    L. Deng, G. Li, S. Han, L. Shi, and Y . Xie, “Model compression and hardware acceleration for neural networks: A comprehensive survey,” Proceedings of the IEEE , vol. 108, no. 4, pp. 485–532, Aug. 2020

    2020

  10. [18]

    Federated learning: Challenges, methods, and future directions,

    T. Li, A. K. Sahu, A. Talwalkar, and V . Smith, “Federated learning: Challenges, methods, and future directions,” IEEE signal processing magazine, vol. 37, no. 3, pp. 50–60, Dec. 2020

    2020

  11. [19]

    Efficient acceleration of deep learning inference on resource-constrained edge devices: A review,

    M. M. H. Shuvo, S. K. Islam, J. Cheng, and B. I. Morshed, “Efficient acceleration of deep learning inference on resource-constrained edge devices: A review,” Proceedings of the IEEE , vol. 111, no. 4, pp. 42– 91, Jan. 2023

    2023

  12. [20]

    Toward tailored models on private aiot devices: Federated direct neural architecture search,

    C. Zhang, X. Yuan, Q. Zhang, G. Zhu, L. Cheng, and N. Zhang, “Toward tailored models on private aiot devices: Federated direct neural architecture search,” IEEE Internet of Things Journal , vol. 9, no. 18, pp. 17 309–17 322, Sep. 2022

    2022

  13. [21]

    Compressing large-scale transformer-based models: A case study on bert,

    P. Ganesh, Y . Chen, X. Lou, M. A. Khan, Y . Yang, H. Sajjad, P. Nakov, D. Chen, and M. Winslett, “Compressing large-scale transformer-based models: A case study on bert,” Transactions of the Association for Computational Linguistics , vol. 9, pp. 1061–1080, Sep. 2021

    2021

  14. [22]

    Efficient transformers: A survey,

    Y . Tay, M. Dehghani, D. Bahri, and D. Metzler, “Efficient transformers: A survey,”ACM Computing Surveys, vol. 55, no. 6, pp. 1–28, Dec. 2022

    2022

  15. [23]

    Pruning deep neural networks by opti- mal brain damage,

    C. Liu, Z. Zhang, and D. Wang, “Pruning deep neural networks by opti- mal brain damage,” in Proc. 15th Annual Conference of the International Speech Communication Association (INTERSPEECH) . Singapore: ISCA, Sep. 2014, pp. 1092–1095

    2014

  16. [24]

    An efficient framework for counting pedestrians crossing a line using low-cost devices: the benefits of distilling the knowledge in a neural network,

    Y . Lin, C. Wang, C. Chang, and H. Sun, “An efficient framework for counting pedestrians crossing a line using low-cost devices: the benefits of distilling the knowledge in a neural network,” Multimedia Tools and Applications, vol. 80, no. 3, pp. 4037–4051, Sep. 2021

    2021

  17. [25]

    Linformer: Self- attention with linear complexity,

    S. Wang, B. Z. Li, M. Khabsa, H. Fang, and H. Ma, “Linformer: Self- attention with linear complexity,” arXiv:2006.04768, 2020

    2006 arXiv

  18. [26]

    Wakeupnet: A mobile- transformer based framework for end-to-end streaming voice trigger,

    Z. Zhang, T. Farnsworth, S. Lin, and S. Karout, “Wakeupnet: A mobile- transformer based framework for end-to-end streaming voice trigger,” arXiv:2210.02904, 2022

    2022 arXiv

  19. [27]

    From hard to soft: Towards more human-like emotion recognition by modelling the perception uncertainty,

    J. Han, Z. Zhang, M. Schmitt, M. Pantic, and B. Schuller, “From hard to soft: Towards more human-like emotion recognition by modelling the perception uncertainty,” in Proc. 25th ACM international conference on Multimedia (ACM MM) . Mountain View, CA, USA: ACM, June 2017, pp. 890–897

    2017

  20. [30]

    The power of interpolation: Under- standing the effectiveness of SGD in modern over-parametrized learn- ing,

    S. Ma, R. Bassily, and M. Belkin, “The power of interpolation: Under- standing the effectiveness of SGD in modern over-parametrized learn- ing,” in Proc. the 35th International Conference on Machine Learning (ICML). Stockholm, Sweden: PMLR, June 2018, pp. 3331–3340

    2018

  21. [31]

    Multi-head atten- tion for speech emotion recognition with auxiliary learning of gender recognition,

    A. Nediyanchath, P. Paramasivam, and P. Yenigalla, “Multi-head atten- tion for speech emotion recognition with auxiliary learning of gender recognition,” in Proc. 45th IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP) . Barcelona, Spain: IEEE, ...

    2020

  22. [32]

    The INTER- SPEECH 2013 computational paralinguistics challenge: Social signals, conflict, emotion, autism,

    B. Schuller, S. Steidl, A. Batliner, A. Vinciarelli, K. Scherer, F. Ringeval, M. Chetouani, F. Weninger, F. Eyben, E. Marchi et al. , “The INTER- SPEECH 2013 computational paralinguistics challenge: Social signals, conflict, emotion, autism,” in Pro. 14th Annual Conference of ...

    2013

  23. [33]

    Self-attention for speech emotion recognition,

    L. Tarantino, P. N. Garner, and A. Lazaridis, “Self-attention for speech emotion recognition,” in Proc. 20th Annual Conference of the Inter- national Speech Communication Association (INTERSPEECH) . Graz, Austria: ISCA, Sep. 2019, pp. 2578–2582

    2019

  24. [34]

    Speechformer: A hierarchical efficient framework incorporating the characteristics of speech,

    W. Chen, X. Xing, X. Xu, J. Pang, and L. Du, “Speechformer: A hierarchical efficient framework incorporating the characteristics of speech,” in Proc. 23th Annual Conference of the International Speech Communication Association (INTERSPEECH) . Incheon, Korea: ISCA, Sep. 2022, p...

    2022

  25. [35]

    Dwformer: Dynamic window transformer for speech emotion recognition,

    S. Chen, X. Xing, W. Zhang, W. Chen, and X. Xu, “Dwformer: Dynamic window transformer for speech emotion recognition,” inProc. 48th IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). Rhodes Island, Greece: IEEE, June 2023, pp. 1–5

    2023

  26. [36]

    A fine-tuned wav2vec 2.0/hubert benchmark for speech emotion recognition, speaker verification and spoken language understanding,

    Y . Wang, A. Boumadane, and A. Heba, “A fine-tuned wav2vec 2.0/hubert benchmark for speech emotion recognition, speaker verification and spoken language understanding,” arXiv:2111.02735, 2021

    2021 arXiv

  27. [37]

    Are sixteen heads really better than one?

    P. Michel, O. Levy, and G. Neubig, “Are sixteen heads really better than one?” in Proc. 33th Annual Conference on Neural Information Processing Systems (NeuIPS) . Vancouver, BC, Canada: MIT Press, Dec. 2019, pp. 14 014–14 024

    2019

  28. [38]

    Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned,

    E. V oita, D. Talbot, F. Moiseev, R. Sennrich, and I. Titov, “Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned,” in Proc. 57th Conference of the Association for Computational Linguistics (ACL) . Florence, Italy: ACL, July 2019...

    2019

  29. [39]

    When BERT plays the lottery, all tickets are winning,

    S. Prasanna, A. Rogers, and A. Rumshisky, “When BERT plays the lottery, all tickets are winning,” in Proc. 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) . Virtual: ACL, July 2020, pp. 3208–3229

    2020

  30. [40]

    Earlybert: Efficient bert training via early-bird lottery tickets,

    X. Chen, Y . Cheng, S. Wang, Z. Gan, Z. Wang, and J. Liu, “Earlybert: Efficient bert training via early-bird lottery tickets,” in Proc. 59th Conference of the Association for Computational Linguistics (ACL) . Virtual: ACL, August 2021, pp. 2195–2207

    2021

  31. [41]

    Reducing transformer depth on demand with structured dropout,

    A. Fan, E. Grave, and A. Joulin, “Reducing transformer depth on demand with structured dropout,” arXiv:1909.11556, 2019

    1909 arXiv

  32. [42]

    On the effect of dropping layers of pre-trained transformer models,

    H. Sajjad, F. Dalvi, N. Durrani, and P. Nakov, “On the effect of dropping layers of pre-trained transformer models,” Computer Speech and Language , vol. 77, p. 101429, July 2023. 14

    2023

  33. [43]

    Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter,

    V . Sanh, L. Debut, J. Chaumond, and T. Wolf, “Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter,” arXiv:1910.01108, 2019

    1910 arXiv

  34. [44]

    Tinybert: Distilling BERT for natural language understanding,

    X. Jiao, Y . Yin, L. Shang, X. Jiang, X. Chen, L. Li, F. Wang, and Q. Liu, “Tinybert: Distilling BERT for natural language understanding,” in Proc. 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Virtual: ACL, July 2020, pp. 4163–4174

    2020

  35. [45]

    Mobilebert: a compact task-agnostic BERT for resource-limited devices,

    Z. Sun, H. Yu, X. Song, R. Liu, Y . Yang, and D. Zhou, “Mobilebert: a compact task-agnostic BERT for resource-limited devices,” in Proc. 58th Conference of the Association for Computational Linguistics (ACL) . Virtual: ACL, July 2020, pp. 2158–2170

    2020

  36. [46]

    Distilhubert: Speech representation learning by layer-wise distillation of hidden-unit bert,

    H. Chang, S. Yang, and H. Lee, “Distilhubert: Speech representation learning by layer-wise distillation of hidden-unit bert,” in Proc. 47th IEEE International Conference on Acoustics, Speech, and Signal Pro- cessing (ICASSP) . Singapore: IEEE, May 2022, pp. 7087–7091

    2022

  37. [47]

    Fmmformer: Efficient and flexible transformer via decomposed near- field and far-field attention,

    T. M. Nguyen, V . Suliafu, S. J. Osher, L. Chen, and B. Wang, “Fmmformer: Efficient and flexible transformer via decomposed near- field and far-field attention,” in Proc. 35th Annual Conference on Neural Information Processing Systems (NeuIPS) . Virtual: MIT Press, Dec. 2021, ...

    2021

  38. [48]

    Is attention better than matrix decomposition?

    Z. Geng, M. Guo, H. Chen, X. Li, K. Wei, and Z. Lin, “Is attention better than matrix decomposition?” in Proc. 9th International Conference on Learning Representations (ICLR). Virtual: OpenReview.net, May 2021

    2021

  39. [49]

    DRONE: data- aware low-rank compression for large NLP models,

    P. H. Chen, H. Yu, I. S. Dhillon, and C. Hsieh, “DRONE: data- aware low-rank compression for large NLP models,” in Proc. 34th Annual Conference on Neural Information Processing Systems (NeuIPS) . Virtual: MIT Press, Dec. 2021, pp. 29 321–29 334

    2021

  40. [50]

    Lightweight and efficient end-to-end speech recognition using low-rank transformer,

    G. I. Winata, S. Cahyawijaya, Z. Lin, Z. Liu, and P. Fung, “Lightweight and efficient end-to-end speech recognition using low-rank transformer,” in Proc. 45th IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP) . Barcelona, Spain: IEEE, May 2020,...

    2020

  41. [51]

    A t2-tensor-aided multiscale transformer for remaining useful life prediction in iiot,

    L. Ren, Z. Jia, X. Wang, J. Dong, and W. Wang, “A t2-tensor-aided multiscale transformer for remaining useful life prediction in iiot,” IEEE Transactions on Industrial Informatics , vol. 18, no. 11, pp. 8108–8118, Nov. 2022

    2022

  42. [52]

    Expandnets: Linear over- parameterization to train compact convolutional networks,

    S. Guo, J. M. Alvarez, and M. Salzmann, “Expandnets: Linear over- parameterization to train compact convolutional networks,” in Proc. 34th Annual Conference on Neural Information Processing Systems (NeuIPS) . Virtual: MIT Press, Dec. 2020, pp. 1298–1310

    2020

  43. [53]

    Repvgg: Making vgg-style convnets great again,

    X. Ding, X. Zhang, N. Ma, J. Han, G. Ding, and J. Sun, “Repvgg: Making vgg-style convnets great again,” in Proc. 34th IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . Virtual: IEEE, June 2021, pp. 13 733–13 742

    2021

  44. [54]

    Acnet: Strengthening the kernel skeletons for powerful cnn via asymmetric convolution blocks,

    X. Ding, Y . Guo, G. Ding, and J. Han, “Acnet: Strengthening the kernel skeletons for powerful cnn via asymmetric convolution blocks,” in Proc. 32th International Conference on Computer Vision (ICCV) . Seoul, Korea: IEEE, Oct. 2019, pp. 1911–1920

    2019

  45. [55]

    Transformer-transducer: End-to-end speech recognition with self-attention,

    C. Yeh, J. Mahadeokar, K. Kalgaonkar, Y . Wang, D. Le, M. Jain, K. Schubert, C. Fuegen, and M. L. Seltzer, “Transformer-transducer: End-to-end speech recognition with self-attention,” arXiv:1910.12977, 2019

    1910 arXiv

  46. [56]

    Conformer: Convolution-augmented transformer for speech recognition,

    A. Gulati, J. Qin, C. Chiu, N. Parmar, Y . Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y . Wu, and R. Pang, “Conformer: Convolution-augmented transformer for speech recognition,” in Proc. 21st Annual Conference of the International Speech Communication Association (INTERSPEECH) ....

    2020

  47. [57]

    On the optimization of deep net- works: Implicit acceleration by overparameterization,

    S. Arora, N. Cohen, and E. Hazan, “On the optimization of deep net- works: Implicit acceleration by overparameterization,” in Proc. the 35th International Conference on Machine Learning (ICML) . Stockholm, Sweden: PMLR, June 2018, pp. 244–253

    2018

  48. [58]

    Scaling laws for autoregressive generative modeling,

    T. Henighan, J. Kaplan, M. Katz, M. Chen, C. Hesse, J. Jackson, H. Jun, T. B. Brown, P. Dhariwal, S. Grayet al., “Scaling laws for autoregressive generative modeling,” arXiv preprint arXiv:2010.14701 , 2020

    2010 arXiv

  49. [59]

    Scaling laws for neural language models,

    J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,” arXiv preprint arXiv:2001.08361 , 2020

    2001 arXiv

  50. [60]

    Scaling laws for transfer,

    D. Hernandez, J. Kaplan, T. Henighan, and S. McCandlish, “Scaling laws for transfer,” arXiv preprint arXiv:2102.01293 , 2021

    2021 arXiv

  51. [61]

    Iemocap: interactive emotional dyadic motion capture database,

    C. Busso, M. Bulut, C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, “Iemocap: interactive emotional dyadic motion capture database,” Language Resources and Evaluation , vol. 42, no. 4, pp. 335–359, Nov. 2008

    2008

  52. [62]

    M3ed: Multi-modal multi-scene multi-label emotional dialogue database,

    J. Zhao, T. Zhang, J. Hu, Y . Liu, Q. Jin, X. Wang, and H. Li, “M3ed: Multi-modal multi-scene multi-label emotional dialogue database,” in Proc. 60th Conference of the Association for Computational Linguistics (ACL). Dublin, Ireland: ACL, May 2022, pp. 5699–5710

    2022

  53. [63]

    The distress analysis interview corpus of human and computer interviews,

    J. Gratch, R. Artstein, G. M. Lucas, G. Stratou, S. Scherer, A. Nazarian, R. Wood, J. Boberg, D. DeVault, S. Marsella et al. , “The distress analysis interview corpus of human and computer interviews,” in Proc. 9th International Conference on Language Resources and Evaluation ...

    2014

  54. [64]

    Emotion recognition with multimodal transformer fusion framework based on acoustic and lexical information,

    L. Guo, L. Wang, J. Dang, Y . Fu, J. Liu, and S. Ding, “Emotion recognition with multimodal transformer fusion framework based on acoustic and lexical information,” IEEE MultiMedia , vol. 29, no. 2, pp. 94–103, Apr. 2022

    2022

  55. [65]

    Lr-gcn: Latent relation-aware graph convolutional network for conversational emotion recognition,

    M. Ren, X. Huang, W. Li, D. Song, and W. Nie, “Lr-gcn: Latent relation-aware graph convolutional network for conversational emotion recognition,” IEEE Transactions on Multimedia, vol. 24, pp. 4422–4432, Oct. 2021

    2021

  56. [66]

    Semantic alignment network for multi-modal emotion recognition,

    M. Hou, Z. Zhang, C. Liu, and G. Lu, “Semantic alignment network for multi-modal emotion recognition,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 33, no. 9, pp. 5318–5329, Feb. 2023

    2023

  57. [67]

    Learning and generalization in overparameterized neural networks,

    Z. Allen-Zhu, Y . Li, and Y . Liang, “Learning and generalization in overparameterized neural networks,” in Proc. 33th Annual Conference on Neural Information Processing Systems (NeuIPS) . Vancouver, BC, Canada: MIT Press, Dec. 2019, pp. 6155–6166

    2019

  58. [68]

    A convergence theory for deep learn- ing via over-parameterization,

    Z. Allen-Zhu, Y . Li, and Z. Song, “A convergence theory for deep learn- ing via over-parameterization,” in Proc. 36th International Conference on Machine Learning (ICML) . Long Beach, California, USA: PMLR, June 2019, pp. 242–252

    2019

  59. [69]

    Representation learning with spectro-temporal-channel attention for speech emotion recognition,

    L. Guo, L. Wang, C. Xu, J. Dang, E. S. Chng, and H. Li, “Representation learning with spectro-temporal-channel attention for speech emotion recognition,” in Proc. 46th IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP) . Toronto, ON, Canada: IEE...

    2021

  60. [70]

    Exploiting vocal tract coordination using dilated cnns for depression detection in naturalistic environments,

    Z. Huang, J. Epps, and D. Joachim, “Exploiting vocal tract coordination using dilated cnns for depression detection in naturalistic environments,” in Proc. 45th IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP) . Barcelona, Spain: IEEE, May 202...

    2020

  61. [71]

    Towards robust deep neural networks for affect and depres- sion recognition from speech,

    A. Othmani, D. Kadoch, K. Bentounes, E. Rejaibi, R. Alfred, and A. Hadid, “Towards robust deep neural networks for affect and depres- sion recognition from speech,” in Proc. 25th International Conference on Pattern Recognition (ICPR). ICPR International Workshops and Challenge...

    2021

  62. [72]

    Hybrid cnn-svm classifier for efficient depression detection system,

    A. Saidi, S. B. Othman, and S. B. Saoud, “Hybrid cnn-svm classifier for efficient depression detection system,” in Proc. 4th International Con- ference on Advanced Systems and Emergent Technologies (IC ASET). Hammamet, Tunisia: IEEE, Dec 2020, pp. 229–234

    2020

  63. [73]

    The detection of depression using multimodal models based on text and voice quality features,

    H. Solieman and E. A. Pustozerov, “The detection of depression using multimodal models based on text and voice quality features,” in Proc. 2021 IEEE Conference of Russian Young Researchers in Electrical and Electronic Engineering (ElConRus) . St. Petersburg and Moscow, Russia:...

    2021

  64. [74]

    Double trouble in double descent: Bias and variance(s) in the lazy regime,

    S. d’Ascoli, M. Refinetti, G. Biroli, and F. Krzakala, “Double trouble in double descent: Bias and variance(s) in the lazy regime,” in Proc. 37th International Conference on Machine Learning (ICML) . Virtual: PMLR, July 2020, pp. 2280–2290

    2020

  65. [75]

    Deep double descent: Where bigger models and more data hurt,

    P. Nakkiran, G. Kaplun, Y . Bansal, T. Yang, B. Barak, and I. Sutskever, “Deep double descent: Where bigger models and more data hurt,” in Proc. 8th International Conference on Learning Representations (ICLR) . Addis Ababa, Ethiopia: OpenReview.net, Apr. 2020

    2020

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.