REVIEW 3 major objections 1 minor 41 references
Adjusting ADC bit proportions in memristor layers for positional encodings reduces execution degradation by about 50 percent while keeping energy consumption stable.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.3
2026-06-27 07:06 UTC pith:GEJR5KBQ
load-bearing objection The paper links positional encoding outputs to ADC degradation on memristor hardware for ASR and reports 30-50% relative fixes via bit reallocation or transform removal, but the abstract gives no isolation data to back the attribution. the 3 major comments →
Positional Encoding in the Context of Memristor-Based Analog Computation for Automatic Speech Recognition
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Large output values of transformed positional encodings cause major degradation within analog-to-digital conversion as part of memristor-based computation. By adjusting the proportion of weight and precision bits of the ADC of specific memristor layers, the degradation of the execution is reduced by ~50% relative, while keeping the estimated energy consumption stable. In scenarios where the ADC cannot be modified, the degradation can be reduced by ~30% relative after removing encoding-related linear transformations.
What carries the argument
The large output values produced by transformed positional encodings when fed into memristor analog vector-matrix multiplications, which overload ADC precision.
Load-bearing premise
The major degradation in ADC is caused by large output values of transformed positional encodings, and targeted bit adjustments or removal of transformations can be made without introducing new unaccounted distortions or energy costs.
What would settle it
Running the speech recognition model on memristor hardware or simulation, measuring ADC input ranges from positional encodings, applying the bit proportion changes to affected layers, and checking whether measured degradation drops by the stated percentages without added energy or new errors.
If this is right
- Adjusting ADC bit proportions in specific layers halves relative degradation from positional encodings.
- Energy consumption estimates stay stable under these ADC adjustments.
- Removing encoding-related linear transformations reduces degradation by ~30% relative when ADC cannot be modified.
- Positional encodings are identified as a primary source of distortion in this analog computation setup for ASR.
Where Pith is reading between the lines
- Layer-specific ADC configurations could be applied to other transformer components beyond positional encodings in memristor systems.
- Hardware designs for sequence models might incorporate type-aware bit allocation to improve overall analog accuracy.
- The method could support deployment of attention-based speech models on low-power analog devices by mitigating encoding-induced errors.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript identifies large output values from transformed positional encodings as causing major degradation in the analog-to-digital conversion (ADC) step during memristor-based vector-matrix multiplication for automatic speech recognition. It proposes two hardware-aware mitigations: reallocating the proportion of weight and precision bits in the ADC of specific layers to achieve an approximately 50% relative reduction in degradation while keeping estimated energy consumption stable, and removing encoding-related linear transformations to achieve an approximately 30% relative reduction when ADC modification is infeasible.
Significance. If the attribution of degradation to positional encoding outputs and the effectiveness of the proposed mitigations are substantiated with detailed experiments, this could offer a targeted approach for adapting transformer components to analog hardware constraints in resource-efficient ASR systems. The work highlights a practical interaction between model architecture choices and device-level noise sources without apparent energy trade-offs.
major comments (3)
- [Abstract] Abstract: The claims of ~50% and ~30% relative reductions in degradation are stated without any reference to the underlying experimental setup, including the specific ASR model, dataset, evaluation metric, number of trials, or statistical measures such as error bars or variance across runs.
- [Abstract] Abstract: The central attribution of ADC degradation specifically to large outputs of transformed positional encodings is not supported by isolating evidence such as per-layer activation histograms, ablation experiments that remove only the positional encoding path, or comparisons against other potential sources of large activations or analog noise (e.g., weight programming variability).
- [Abstract] Abstract: The assertion that energy consumption remains stable after ADC bit reallocation or transform removal lacks supporting details on the energy model, including how changes in precision bits affect dynamic range, quantization noise, or total power draw, which could offset the reported gains.
minor comments (1)
- The abstract would benefit from a concise statement of the base neural architecture and memristor parameters used to ground the quantitative claims.
Simulated Author's Rebuttal
We thank the referee for the detailed feedback on our manuscript. We address each major comment point-by-point below, proposing targeted revisions to the abstract to improve clarity while preserving the manuscript's core contributions.
read point-by-point responses
-
Referee: [Abstract] Abstract: The claims of ~50% and ~30% relative reductions in degradation are stated without any reference to the underlying experimental setup, including the specific ASR model, dataset, evaluation metric, number of trials, or statistical measures such as error bars or variance across runs.
Authors: We agree that the abstract would benefit from explicit references to the experimental setup. In the revised manuscript, we will modify the abstract to include the specific ASR model, dataset (such as LibriSpeech), evaluation metric (word error rate), and note that results are from multiple trials with variance reported. The detailed setup is described in the methods and results sections of the manuscript. revision: yes
-
Referee: [Abstract] Abstract: The central attribution of ADC degradation specifically to large outputs of transformed positional encodings is not supported by isolating evidence such as per-layer activation histograms, ablation experiments that remove only the positional encoding path, or comparisons against other potential sources of large activations or analog noise (e.g., weight programming variability).
Authors: The manuscript includes per-layer activation histograms and ablation studies that isolate the positional encoding contributions, as detailed in the results section. To address this comment, we will revise the abstract to briefly reference that the attribution is based on such isolating analyses. We believe this provides the necessary support without misrepresenting the work. revision: yes
-
Referee: [Abstract] Abstract: The assertion that energy consumption remains stable after ADC bit reallocation or transform removal lacks supporting details on the energy model, including how changes in precision bits affect dynamic range, quantization noise, or total power draw, which could offset the reported gains.
Authors: The energy model and its accounting for dynamic range, quantization noise, and power draw are explained in the methods section. We will update the abstract to include a concise reference to the stability of energy estimates under the proposed changes. This revision will clarify the claim. revision: yes
Circularity Check
No circularity: empirical hardware adjustments with no derivation chain or self-referential predictions
full rationale
The paper reports an empirical observation (large positional-encoding outputs degrade ADC in memristor layers) followed by direct experimental mitigations (reallocating ADC weight/precision bits in specific layers or removing linear transforms). These yield measured ~50% or ~30% relative degradation reductions at stable energy. No mathematical derivation, first-principles prediction, fitted-parameter renaming, or self-citation load-bearing step is present in the abstract or described claims. The central result is an experimental outcome, not a quantity forced by construction from its own inputs. No equations or uniqueness theorems are invoked that reduce to prior self-referential statements.
Axiom & Free-Parameter Ledger
axioms (1)
- domain assumption Memristors enable analog vector-matrix-multiplication subject to larger distortion in weight programming and execution.
read the original abstract
Memristors provide a new chance for resource-efficient computation of neural models for natural language processing by enabling analog execution of vector-matrix-multiplication. Yet, computations on these devices are currently subject to larger distortion, both in weight programming and execution. In this work, we identify large output values of transformed positional encodings to cause major degradation within analog-to-digital conversion (ADC) as part of memristor-based computation. By adjusting the proportion of weight and precision bits of the ADC of specific memristor layers, we reduce the degradation of the execution by ~50% relative, while keeping the estimated energy consumption stable. Additionally, we investigate scenarios where the ADC cannot be modified. In that case the degradation can be reduced by ~30% relative after removing encoding-related linear transformations.
Reference graph
Works this paper leans on
-
[1]
Introduction The increase in model size of modern day natural language processing models pushes the need for efficient computing. One family of device types that offer efficient computation of vector-matrix-multiplication (VMM) based operations is so- called memristors. With physically programmable resistance states within a crossbar array, such devices c...
work page internal anchor Pith review Pith/arXiv arXiv 2026
-
[2]
Memristors Memristors are physical circuit elements that change their resis- tance through the application of a sufficiently large voltage or an electric field [20]
Background 2.1. Memristors Memristors are physical circuit elements that change their resis- tance through the application of a sufficiently large voltage or an electric field [20]. The resistance states within the device are non-volatile, meaning they are held until the next programming cycle. The major benefit of memristors for machine learning is the c...
-
[3]
We conduct the majority of our studies on LibriSpeech
Experimental Setup We make use of a Conformer [12] ASR system with Connectionist-Temporal-Classification (CTC)-based [27] out- puts. We conduct the majority of our studies on LibriSpeech
-
[4]
For LibriSpeech, we use ARPA-phonemes from the provided lexicon as target labels
and verify our findings on Loquacious as a second task us- ing the 250 hour subset for training. For LibriSpeech, we use ARPA-phonemes from the provided lexicon as target labels. For Loquacious, we use byte-pair-encoding [29] with 128 merges. As the memristor simulation requires a substantial amount of computation, we limit our evaluations to thedev-other...
2048
-
[5]
Baseline For our first comparison, we run the pipeline for models with and without PEs
Experiments 4.1. Baseline For our first comparison, we run the pipeline for models with and without PEs. The results can be seen in Table 1. For the 3https://github.com/rwth-i6/SynaptogenML Table 1:Results comparing baseline recognition and memristor execution, ADC is configured to 4-bit precision and 4-bit range. Activations are quantized to 8-bit. Weigh...
-
[6]
We identified the encodings to require specific care, as they do not integrate into the memris- tor execution as smoothly as the rest of the network
Conclusion In this work, we investigated the issues that can arise when map- ping a state-of-the-art Conformer model with relative PE onto simulated memristor devices. We identified the encodings to require specific care, as they do not integrate into the memris- tor execution as smoothly as the rest of the network. In order to mitigate the degradation ca...
-
[7]
Clusters4Future
Acknowledgments This work was partially supported by NeuroSys, which as part of the initiative “Clusters4Future” is funded by the Federal Ministry of Research, Technology and Space BMFTR (fund- ing IDs 03ZU2106DA and 03ZU2106DD), and by the project RESCALE within the programAI Lighthouse Projects for the Environment, Climate, Nature and Resourcesfunded by...
-
[8]
Generative AI Use Disclosure Generative AI was used in this work for proofreading and gram- mar correction, but was not used to generate content
-
[9]
Relia- bility Aspects,
D. J. Wouters, Y .-Y . Chen, A. Fantini, and N. Raghavan, “Relia- bility Aspects,” inResistive Switching. John Wiley & Sons, Ltd, 2016, ch. 21, pp. 597–622
2016
-
[10]
A compute-in-memory chip based on resistive random-access memory,
W. Wan, R. Kubendran, C. Schaefer, S. B. Eryilmaz, W. Zhang, D. Wu, S. Deiss, P. Raina, H. Qian, B. Gao, S. Joshi, H. Wu, H.- S. P. Wong, and G. Cauwenberghs, “A compute-in-memory chip based on resistive random-access memory,”Nature, vol. 608, no. 7923, pp. 504–512, Aug. 2022
2022
-
[11]
Memristor-based hardware accelerators for artificial in- telligence,
Y . Huang, T. Ando, A. Sebastian, M.-F. Chang, J. J. Yang, and Q. Xia, “Memristor-based hardware accelerators for artificial in- telligence,”Nature Reviews Electrical Engineering, vol. 1, no. 5, pp. 286–299, May 2024
2024
-
[12]
High-density integration for RRAM-based computing-in-memory,
Y . He, C. Ma, Z. Jia, Y . Su, and J. Tang, “High-density integration for RRAM-based computing-in-memory,”Nature Reviews Elec- trical Engineering, Jan. 2026
2026
-
[13]
RxNN: A Framework for Evaluating Deep Neural Networks on Resistive Crossbars,
S. Jain, A. Sengupta, K. Roy, and A. Raghunathan, “RxNN: A Framework for Evaluating Deep Neural Networks on Resistive Crossbars,”IEEE Transactions on Computer-Aided Design of In- tegrated Circuits and Systems, vol. 40, no. 2, pp. 326–338, 2021
2021
-
[14]
Fusion of memristor and digital compute-in-memory processing for energy- efficient edge computing,
T.-H. Wen, J.-M. Hung, W.-H. Huang, C.-J. Jhang, Y .-C. Lo, H.- H. Hsu, Z.-E. Ke, Y .-C. Chen, Y .-H. Chin, C.-I. Su, W.-S. Khwa, C.-C. Lo, R.-S. Liu, C.-C. Hsieh, K.-T. Tang, M.-S. Ho, C.-C. Chou, Y .-D. Chih, T.-Y . J. Chang, and M.-F. Chang, “Fusion of memristor and digital compute-in-memory processing for energy- efficient edge computing,”Science, vol...
2024
-
[15]
J. Souto, G. Botella, D. Garc ´ıa, R. Murillo, and A. del Bar- rio, “Neuromorphic Circuit Simulation with Memristors: Design and Evaluation Using MemTorch for MNIST and CIFAR,”arXiv preprint arXiv:2407.13410, 2024
-
[16]
Running Conventional Automatic Speech Recogni- tion on Memristor Hardware: A Simulated Approach,
N. Rossenbach, B. Hilmes, L. Brackmann, M. Gunz, and R. Schl¨uter, “Running Conventional Automatic Speech Recogni- tion on Memristor Hardware: A Simulated Approach,” inInter- speech, 2025, pp. 2560–2564
2025
-
[17]
Attention is All you Need,
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is All you Need,” inAdvances in Neural Information Processing Systems, vol. 30, 2017
2017
-
[18]
Self-Attention with Rela- tive Position Representations,
P. Shaw, J. Uszkoreit, and A. Vaswani, “Self-Attention with Rela- tive Position Representations,” inProceedings of the 2018 Confer- ence of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume 2 (Short Papers), 2018, pp. 464–468
2018
-
[19]
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context,
Z. Dai, Z. Yang, Y . Yang, J. Carbonell, Q. Le, and R. Salakhut- dinov, “Transformer-XL: Attentive Language Models beyond a Fixed-Length Context,” inProceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, A. Korho- nen, D. Traum, and L. M `arquez, Eds. Florence, Italy: Associa- tion for Computational Linguistics, Jul. 20...
2019
-
[20]
Conformer: Convolution-augmented Transformer for Speech Recognition,
A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y . Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y . Wu, and R. Pang, “Conformer: Convolution-augmented Transformer for Speech Recognition,” in Interspeech, 2020, pp. 5036–5040
2020
-
[21]
An Investigation of Positional Encoding in Transformer-based End-to-end Speech Recognition,
F. Yue and T. Ko, “An Investigation of Positional Encoding in Transformer-based End-to-end Speech Recognition,” in12th In- ternational Symposium on Chinese Spoken Language Processing (ISCSLP), 2021, pp. 1–5
2021
-
[22]
A Transformer with Interleaved Self-attention and Convolution for Hybrid Acoustic Models,
L. Lu, “A Transformer with Interleaved Self-attention and Convolution for Hybrid Acoustic Models,”arXiv preprint arXiv:1910.10352, 2019
-
[23]
A Defect-Tolerant Reusable Network of DACs for Wafer-Scale Integration,
N. Laflamme-Mayer, G. Kowarzyk, Y . Blaqui`ere, Y . Savaria, and M. Sawan, “A Defect-Tolerant Reusable Network of DACs for Wafer-Scale Integration,”IEEE Transactions on Very Large Scale Integration (VLSI) Systems, vol. 27, no. 2, pp. 304–315, 2019
2019
-
[24]
Synaptogen: A Cross- Domain Generative Device Model for Large-Scale Neuromorphic Circuit Design,
T. Hennen, L. Brackmann, T. Ziegler, S. Siegel, S. Menzel, R. Waser, D. J. Wouters, and D. Bedau, “Synaptogen: A Cross- Domain Generative Device Model for Large-Scale Neuromorphic Circuit Design,”IEEE Transactions on Electron Devices, vol. 71, no. 9, pp. 5345–5353, 2024
2024
-
[25]
Sisyphus, a workflow manager de- signed for machine translation and automatic speech recognition,
J. Peter, E. Beck, and H. Ney, “Sisyphus, a workflow manager de- signed for machine translation and automatic speech recognition,” inEMNLP 2018: System Demonstrations, Brussels, Belgium, Oc- tober 31 - November 4, 2018, pp. 84–89
2018
-
[26]
Returnn: The RWTH extensible training framework for universal recurrent neural networks,
P. Doetsch, A. Zeyer, P. V oigtlaender, I. Kulikov, R. Schl¨uter, and H. Ney, “Returnn: The RWTH extensible training framework for universal recurrent neural networks,” inICASSP 2017, New Or- leans, LA, USA, March 5-9, 2017, pp. 5345–5349
2017
-
[27]
RASR/NN: The RWTH neural network toolkit for speech recog- nition,
S. Wiesler, A. Richard, P. Golik, R. Schl ¨uter, and H. Ney, “RASR/NN: The RWTH neural network toolkit for speech recog- nition,” in2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, May 2014
2014
-
[28]
The missing memristor found,
D. B. Strukov, G. S. Snider, D. R. Stewart, and R. S. Williams, “The missing memristor found,”Nature, vol. 453, no. 7191, pp. 80–83, May 2008
2008
-
[29]
Analogue signal and image processing with large memristor crossbars,
C. Li, M. Hu, H. Li, Yunningand Jiang, N. Ge, E. Montgomery, J. Zhang, W. Song, N. D´avila, C. E. Graves, Z. Li, J. P. Strachan, P. Lin, Z. Wang, M. Barnell, Q. Wu, R. S. Williams, J. J. Yang, and Q. Xia, “Analogue signal and image processing with large memristor crossbars,”Nature Electronics, vol. 1, no. 1, pp. 52– 59, 2018
2018
-
[30]
Intrinsic Bounds for Computing Precision in Memristor-Based Vector-by-Matrix Multipliers,
M. R. Mahmoodi, A. F. Vincent, H. Nili, and D. B. Strukov, “Intrinsic Bounds for Computing Precision in Memristor-Based Vector-by-Matrix Multipliers,”IEEE Transactions on Nanotech- nology, vol. 19, pp. 429–435, 2020
2020
-
[31]
Experimental Investi- gation of 4-kb RRAM Arrays Programming Conditions Suitable for TCAM,
A. Grossi, E. Vianello, C. Zambelli, P. Royer, J.-P. Noel, B. Gi- raud, L. Perniola, P. Olivo, and E. Nowak, “Experimental Investi- gation of 4-kb RRAM Arrays Programming Conditions Suitable for TCAM,”IEEE Transactions on Very Large Scale Integration (VLSI) Systems, vol. 26, no. 12, pp. 2599–2607, 2018
2018
-
[32]
MNIST handwritten digit database,
Y . LeCun and C. Cortes, “MNIST handwritten digit database,” 2010
2010
-
[33]
Learning Multiple Layers of Features from Tiny Images,
A. Krizhevsky, “Learning Multiple Layers of Features from Tiny Images,” Tech. Rep., 2009
2009
-
[34]
Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
P. Warden, “Speech Commands: A Dataset for Limited-V ocabulary Speech Recognition,”arXiv preprint arXiv:1804.03209, Apr. 2018
work page internal anchor Pith review Pith/arXiv arXiv 2018
-
[35]
Con- nectionist Temporal Classification: Labelling Unsegmented Se- quence Data with Recurrent Neural Networks,
A. Graves, S. Fern ´andez, F. Gomez, and J. Schmidhuber, “Con- nectionist Temporal Classification: Labelling Unsegmented Se- quence Data with Recurrent Neural Networks,” inProceedings of the 23rd International Conference on Machine Learning, ser. ICML ’06. New York, NY , USA: Association for Computing Machinery, 2006, p. 369–376
2006
-
[36]
Lib- rispeech: An ASR Corpus Based on Public Domain Audio Books,
V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Lib- rispeech: An ASR Corpus Based on Public Domain Audio Books,” inIEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015, pp. 5206–5210
2015
-
[37]
Neural Machine Trans- lation of Rare Words with Subword Units,
R. Sennrich, B. Haddow, and A. Birch, “Neural Machine Trans- lation of Rare Words with Subword Units,” inProceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers). Association for Computa- tional Linguistics, Aug. 2016, pp. 1715–1725
2016
-
[38]
KenLM: Faster and Smaller Language Model Queries,
K. Heafield, “KenLM: Faster and Smaller Language Model Queries,” inProceedings of the Sixth Workshop on Statistical Ma- chine Translation. Association for Computational Linguistics, Jul. 2011, pp. 187–197
2011
-
[39]
N. Rossenbach, R. Schmitt, T. Raissi, S. Berger, L. Kleppel, and R. Schl ¨uter, “Supplementary Resources and Analysis for Auto- matic Speech Recognition Systems Trained on the Loquacious Dataset,”arXiv preprint arXiv:2512.17915, 2025
-
[40]
On the Variance of the Adaptive Learning Rate and Beyond,
L. Liu, H. Jiang, P. He, W. Chen, X. Liu, J. Gao, and J. Han, “On the Variance of the Adaptive Learning Rate and Beyond,” in Proceedings of the Eighth International Conference on Learning Representations (ICLR), Apr. 2020
2020
-
[41]
SpecAugment: A Simple Data Augmenta- tion Method for Automatic Speech Recognition,
D. S. Park, W. Chan, Y . Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V . Le, “SpecAugment: A Simple Data Augmenta- tion Method for Automatic Speech Recognition,” inInterspeech, 2019, pp. 2613–2617
2019
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.