REVIEW 3 major objections 7 minor 23 references
Uncertainty Quantification for Motor Imagery BCI -- Machine Learning vs. Deep Learning
T0 review · 3 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper claims that classical BCI classifiers (CSP-LDA and MDRM-T) provide better-calibrated confidence estimates than deep learning methods, while deep learning still gives higher classification accuracy.
desk verdict Useful BCI-specific comparison, but the headline 'classical beats deep' ranking rides on an asymmetric calibration setup and missing significance tests. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism carrying the comparison is MDRM-T, a temperature-scaled version of the Riemannian minimum-distance classifier. MDRM represents each EEG epoch by its channel covariance matrix, measures Riemannian distance $d_i$ to each class mean, and the paper converts distances to probabilities with $\hat{y}_i = \exp(-d_i^2/T)/\sum_j \exp(-d_j^2/T)$, where $T$ is fit after training to minimize calibration error on the training data. CSP-LDA supplies probabilities by assuming Gaussian class densities under Bayesian LDA. The evaluation machinery is Expected Calibration Error and Net Calibration Error, which bin predictions by confidence and compare bin accuracy to bin confidence, plus Brier score and rejection-accuracy curves.
What would settle it
Take the four datasets, run repeated 80-20 within-subject splits, and compute per-subject paired differences in ECE between CSP-LDA/MDRM-T and the best deep model; if the paired differences straddle zero at conventional significance levels, the calibration ranking in Table 1 is not robust. A second test: inject artifact or out-of-distribution EEG epochs and check whether CSP-LDA and MDRM-T remain better calibrated than DUQ and Deep Ensembles, since the paper's limitations section notes no out-of-distribution evaluation was run.
Extended reading notes
Core claim
On the paper's own terms, the discovery is a two-sided separation: uncertainty calibration and classification accuracy do not go together in current motor-imagery models. CSP-LDA and the temperature-scaled MDRM-T produce confidence values that most closely track true accuracy, with the lowest expected calibration error and NCE near zero, whereas every deep-learning method tested — DUQ, Deep Ensembles, and a standard CNN — is overconfident. Deep Ensembles and CNNs nevertheless give the highest classification accuracies and best Brier scores on most datasets. The paper additionally shows that raw MDRM is systematically underconfident, and that attaching temperature scaling fixes this without changing any classification, since only the confidence scale is adjusted.
Load-bearing premise
The load-bearing premise is that a single 80-20 within-subject split, summarized as means with standard deviations across subjects, is enough to rank the methods; no significance tests are reported, and several ECE differences are smaller than the subject-to-subject spread (for example, 0.231 ± 0.116 versus 0.276 ± 0.115 on the Steyrl dataset).
Editorial extensions
If this is right
- Selective rejection raises accuracy for every model on every dataset, so a BCI can trade coverage for reliability by declining ambiguous trials.
- CSP-LDA and MDRM-T provide confidence values that can be read literally for in-distribution motor-imagery data; deep methods' probabilities should be interpreted as rankings, not true likelihoods.
- MDRM-T obtains the same accuracy as MDRM with better calibration, so temperature scaling adds uncertainty quality without sacrificing classification.
- Non-deep pipelines train in under a second and infer in under 0.1 ms per sample, making them feasible for online iterative BCI use, while deep nets cost orders of magnitude more.
- The ECE/NCE/Brier/rejection evaluation protocol is dataset-agnostic and can be applied to future motor-imagery classifiers to certify their probabilities.
Reading between the lines
- A natural next step the paper does not run is temperature scaling on the deep models; since DUQ and Deep Ensembles are overconfident in the same direction, a fitted temperature could close most of the calibration gap and combine deep accuracy with trustworthy probabilities.
- The rejection results imply a practical design rule: a BCI can operate in a 'say nothing when unsure' mode, and the reliable probability scale from CSP-LDA or MDRM-T lets the threshold be set from calibration curves rather than heuristics.
- If the ranking transfers to out-of-distribution data — which the paper explicitly leaves untested — then safety-critical BCI applications should prefer the non-deep models even when deep models show higher accuracy.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper compares uncertainty quantification (UQ) for motor-imagery brain-computer interfaces between classical machine learning classifiers (CSP-LDA, MDRM, and a proposed temperature-scaled MDRM-T) and deep learning methods (ShallowConvNet CNN, Deep Ensembles, and DUQ) on four public datasets accessed through MOABB. Using ECE, NCE, Brier score, calibration plots, and rejection-accuracy curves, it reports that the classical methods are better calibrated, while Deep Ensembles and CNNs achieve higher classification accuracy, and that all models can improve accuracy by rejecting low-confidence samples. The main contributions are the introduction of MDRM-T and the first systematic comparison of UQ between these model families in motor-imagery BCI.
Significance. The paper addresses a real gap: uncertainty quantification for motor-imagery BCIs has mostly been studied in the deep learning setting, while classical BCI classifiers also output probabilities that are routinely used without calibration checks. The study is useful in its scope, public datasets, standard metrics, and the practical rejection analysis, and it provides a code repository. If the central comparison holds, the practical message that CSP-LDA and MDRM-T probabilities can support selective BCI control would be valuable. The paper does not ship formal proofs or parameter-free derivations, but it does provide a reproducible benchmark setup. The main limitation is that the comparison is not yet apples-to-apples because calibration treatments differ across model families, and the reported rankings lack statistical support.
major comments (3)
- [Section 2.1, Table 1] The central comparison is asymmetric in calibration treatment. MDRM-T is explicitly post-hoc temperature-scaled on the training data to minimize calibration error, and DUQ also converts distances to probabilities with a temperature-scaled softmax as in Eq. (1). The CNN and Deep Ensembles, in contrast, are evaluated through their raw softmax outputs with no equivalent calibration step. Since temperature scaling is the standard remedy for deep learning overconfidence, the lower ECE and NCE reported for MDRM-T and CSP-LDA in Table 1 may reflect the calibration step rather than an inherent property of the model class. To support the abstract claim that CSP-LDA and MDRM-T give the best uncertainty estimates, the manuscript should apply the same temperature-scaling (or an equivalent calibration) procedure to the CNN and Deep Ensembles and report both uncalibrated and calibrated results.
- [Section 3, Table 1] The comparative rankings are based on per-subject means with large between-subject standard deviations, and no significance tests or effect sizes are reported. For example, on the Steyrl dataset the ECE for CSP-LDA is 0.231 ± 0.116, while Deep Ensembles give 0.276 ± 0.115; on the Zhou dataset CSP-LDA gives 0.122 ± 0.042 versus MDRM-T 0.148 ± 0.057. Differences of this size are within one standard deviation in several cases. The claims that MDRM-T and CSP-LDA give the best calibration and that Deep Ensembles and CNNs give the best accuracy need paired statistical tests across subjects within each dataset, such as Wilcoxon signed-rank tests or paired permutation tests, or at least a report of subject-level wins and losses. Without this, the main ranking is not robust.
- [Section 2.2.1, Figure 3] The rejection analysis claims that all models can separate easy from difficult samples, but it does not compare confidence-based rejection against a random-rejection baseline. Since random rejection leaves an expected accuracy equal to the full-set accuracy, the upward slopes in Figure 3 are informative only relative to such a baseline. The paper should add a random-rejection curve or a quantitative summary such as the area under the accuracy-rejection curve; otherwise the claim that the models have useful rejection ability is not fully supported. This is load-bearing for the practical selective-BCI conclusion.
minor comments (7)
- [Section 2.1, Eq. (1)] The displayed equation for temperature scaling is typeset in a confusing way; the denominator and the probability symbol need proper formatting, and it should be stated explicitly whether the temperature T is optimized per subject or per dataset and with what bounds.
- [Section 2.2] The paper uses the term 'confidence' without a formal definition; it should state explicitly that confidence is the maximum predicted softmax probability for a test sample, since this is relevant to both ECE binning and the rejection procedure.
- [Figures 2 and 3] It is not stated whether the calibration and rejection curves are pooled across all subjects or averaged per subject; this should be clarified, because the two choices can produce visually different curves and different interpretations.
- [Table 1] The boldface convention for the best value per row is not consistently visible in the manuscript text; please verify that the boldface is applied to exactly one entry per metric and dataset according to the stated direction of improvement.
- [References] References [7] and [14] appear to refer to the same Schirrmeister et al. paper; please consolidate them into a single citation.
- [Section 2.1] The sentence 'Deep Ensembles [15] uses five ShallowConvNet models' has a subject-verb agreement error; it should be 'use'.
- [Section 2.1] For DUQ, the paper should clarify how the temperature scaling in Eq. (1) is optimized and whether the RBF length scale of 0.2 is tuned or fixed, since both choices affect the DUQ calibration results.
Circularity Check
No significant circularity: MDRM-T temperature scaling is fitted on the training split and evaluated on held-out test data, and the paper's self-citations are background statements, not load-bearing evidence.
full rationale
The central claims are empirical comparisons on four external MOABB datasets with an 80-20 within-subject split. The only parameter fitted to a calibration objective is MDRM-T's temperature T (Section 2.1, Eq. 1), optimized on the training split and then assessed on held-out test data; this is standard post-hoc calibration rather than a circular fit to the reported test ECE/NCE. DUQ also receives temperature scaling, while CNN and Deep Ensembles are evaluated with raw softmax; this is a comparison asymmetry that bears on fairness and correctness risk, but it does not make the ranking true by construction. The self-citations [1] and [3] are used only to state that UQ in MI-BCI has received little attention and that Deep Ensembles are often considered strong UQ baselines; these are background claims and the paper's own empirical results partially contradict them, so the citations are not load-bearing. No uniqueness theorem, ansatz, or definition is imported from the authors' prior work to force the conclusion. The main weakness is statistical: Table 1 reports only means and standard deviations across subjects without significance tests, so the headline ranking may not be robust, but that is a scientific-evidence concern, not circularity.
Assumptions & free parameters
free parameters (4)
- MDRM-T temperature T =
not reported per dataset
- DUQ RBF length scale =
0.2
- DUQ softmax temperature =
not reported
- Number of CSP filters =
8
assumptions (4)
- domain assumption Softmax of negative squared Riemannian distances yields valid class probabilities for MDRM and DUQ.
- domain assumption Binning confidence into 10 bins gives a reliable ECE estimate.
- domain assumption The 80-20 within-subject split is representative of subject-level performance.
- domain assumption MOABB provides comparable preprocessing and data splits across datasets.
Cite this review
Pith. "Pith review of Uncertainty Quantification for Motor Imagery BCI -- Machine Learning vs. Deep Learning." pith.science (2026). https://pith.science/paper/2UAPDQXJ
@misc{pith2026250707511,
author = {Pith},
title = {Pith review of: Uncertainty Quantification for Motor Imagery BCI -- Machine Learning vs. Deep Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/2UAPDQXJ}},
note = {Machine review of arXiv:2507.07511}
}
read the original abstract
Brain-computer interfaces (BCIs) turn brain signals into functionally useful output, but they are not always accurate. A good Machine Learning classifier should be able to indicate how confident it is about a given classification, by giving a probability for its classification. Standard classifiers for Motor Imagery BCIs do give such probabilities, but research on uncertainty quantification has been limited to Deep Learning. We compare the uncertainty quantification ability of established BCI classifiers using Common Spatial Patterns (CSP-LDA) and Riemannian Geometry (MDRM) to specialized methods in Deep Learning (Deep Ensembles and Direct Uncertainty Quantification) as well as standard Convolutional Neural Networks (CNNs). We found that the overconfidence typically seen in Deep Learning is not a problem in CSP-LDA and MDRM. We found that MDRM is underconfident, which we solved by adding Temperature Scaling (MDRM-T). CSP-LDA and MDRM-T give the best uncertainty estimates, but Deep Ensembles and standard CNNs give the best classifications. We show that all models are able to separate between easy and difficult estimates, so that we can increase the accuracy of a Motor Imagery BCI by rejecting samples that are ambiguous.
Reference graph
Works this paper leans on
-
[1]
Uncertainty Quantification for Motor Imagery BCI -- Machine Learning vs. Deep Learning
INTRODUCTION Research on non-invasive BCIs relies on strong Machine Learning algorithms that can accurately detect cognitive states from EEG. In this direction, a lot of work is done to improve the classification accuracy of models under vari- ous circumstances. However, no model will ever be perfect, which makes it desirable to know how confident the mod...
work page Pith review arXiv 2025
-
[2]
Specifically, we use the datasets from Steyrl et al
METHODS In our experiments we use multiple existing datasets from a standard Motor Imagery BCI set-up where participants imagine a cued movement which is then decoded from EEG signals. Specifically, we use the datasets from Steyrl et al. [8], Zhou et al. [9], the BCI Competition IV - dataset 2b [10] and BCI Competition IV - dataset 2a [11]. All datasets w...
-
[3]
RESULTS We first observe the uncertainty calibration for the different models in Figure 2. We see that MDRM is generally under- confident (accuracy higher than confidence), while the Deep Learning methods are all overconfident. CSP-LDA is gen- Table 1. Comparison of within-subject mean performance, with the standard deviation across subjects. Boldface ind...
work page 2021
-
[4]
DISCUSSION We have compared the Uncertainty Quantification capabil- ities of non-Deep Learning methods that are common for Motor Imagery BCIs against Deep Learning methods es- pecially designed to improve uncertainty estimation. We see variations over different datasets, but we generally find that while the Deep Learning methods showed better accu- racies...
-
[5]
CONCLUSION Our paper introduced MDRM-T, where MDRM is combined with temperature scaling to get better calibrated uncertainty estimates. We show that it gives better calibrated uncertainty estimates than MDRM, without affecting the classification performance. We also found that standard BCI models give better uncertainty estimates than Deep Learning models...
-
[6]
Un- certainty quantification for cross-subject motor imagery classification,
Prithviraj Manivannan, Ivo Pascal de Jong, Matias Valdenegro-Toro, and Andreea Ioana Sburlea, “Un- certainty quantification for cross-subject motor imagery classification,” in Graz Brain-Computer Interface Con- ference, 2024, vol. 9, pp. 86–91
work page 2024
-
[7]
Robust mo- tor imagery tasks classification approach using bayesian neural network,
Daily Milan ´es-Hermosilla, Rafael Trujillo-Codorni ´u, Saddid Lamar-Carbonell, Roberto Sagar ´o-Zamora, Jorge Jadid Tamayo-Pacheco, John Jairo Villarejo- Mayor, and Denis Delisle-Rodriguez, “Robust mo- tor imagery tasks classification approach using bayesian neural network,” Sensors, vol. 23, no. 2, pp. 703, 2023
work page 2023
-
[8]
Uncertainty quantification in machine learning for biosignal applications–a review,
Ivo Pascal de Jong, Andreea Ioana Sburlea, and Ma- tias Valdenegro-Toro, “Uncertainty quantification in machine learning for biosignal applications–a review,” arXiv preprint arXiv:2312.09454, 2023
arXiv 2023
Show all 23 references
-
[9]
Opti- mizing spatial filters for robust eeg single-trial analy- sis,
Benjamin Blankertz, Ryota Tomioka, Steven Lemm, Motoaki Kawanabe, and Klaus-Robert Muller, “Opti- mizing spatial filters for robust eeg single-trial analy- sis,” IEEE Signal processing magazine , vol. 25, no. 1, pp. 41–56, 2007
2007
-
[10]
Multiclass brain–computer interface classification by riemannian geometry,
Alexandre Barachant, St ´ephane Bonnet, Marco Con- gedo, and Christian Jutten, “Multiclass brain–computer interface classification by riemannian geometry,” IEEE Transactions on Biomedical Engineering, vol. 59, no. 4, pp. 920–928, 2011
2011
-
[11]
Uncertainty estimation using a single deep deterministic neural network,
Joost Van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal, “Uncertainty estimation using a single deep deterministic neural network,” in International con- ference on machine learning . PMLR, 2020, pp. 9690– 9700
2020
-
[12]
Deep learn- ing with convolutional neural networks for eeg decoding and visualization,
Robin Tibor Schirrmeister, Jost Tobias Springenberg, Lukas Dominique Josef Fiederer, Martin Glasstetter, Katharina Eggensperger, Michael Tangermann, Frank Hutter, Wolfram Burgard, and Tonio Ball, “Deep learn- ing with convolutional neural networks for eeg decoding and visualiz...
2017
-
[13]
Random forests in non-invasive sensorimotor rhythm brain-computer interfaces: a prac- tical and convenient non-linear classifier,
David Steyrl, Reinhold Scherer, Josef Faller, and Ger- not R M ¨uller-Putz, “Random forests in non-invasive sensorimotor rhythm brain-computer interfaces: a prac- tical and convenient non-linear classifier,” Biomedical Engineering/Biomedizinische Technik, vol. 61, no. 1, pp. 7...
2016
-
[14]
A fully automated trial selection method for optimization of motor imagery based brain-computer interface,
Bangyan Zhou, Xiaopei Wu, Zhao Lv, Lei Zhang, and Xiaojin Guo, “A fully automated trial selection method for optimization of motor imagery based brain-computer interface,” PloS one, vol. 11, no. 9, pp. e0162657, 2016
2016
-
[15]
Brain– computer communication: motivation, aim, and impact of exploring a virtual apartment,
Robert Leeb, Felix Lee, Claudia Keinrath, Reinhold Scherer, Horst Bischof, and Gert Pfurtscheller, “Brain– computer communication: motivation, aim, and impact of exploring a virtual apartment,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 15, no. 4,...
2007
-
[16]
Review of the bci competition iv,
Michael Tangermann, Klaus-Robert M ¨uller, Ad Aert- sen, Niels Birbaumer, Christoph Braun, Clemens Brun- ner, Robert Leeb, Carsten Mehring, Kai J Miller, Ger- not R M¨uller-Putz, et al., “Review of the bci competition iv,” Frontiers in neuroscience, vol. 6, pp. 55, 2012
2012
-
[17]
Mother of all bci benchmarks,
Bruno Aristimunha, Igor Carrara, Pierre Guetschel, Sara Sedlar, Pedro Rodrigues, Jan Sosulski, Di- vyesh Narayanan, Erik Bjareholt, Quentin Barthelemy, Robin Tibor Schirrmeister, Reinmar Kobler, Emmanuel Kalunga, Ludovic Darmet, Cattan Gregoire, Ali Ab- dul Hussain, Ramiro Gat...
2025
-
[18]
On calibration of modern neural networks,
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Wein- berger, “On calibration of modern neural networks,” in International conference on machine learning . PMLR, 2017, pp. 1321–1330
2017
-
[19]
Deep learning with convolutional neural net- works for eeg decoding and visualization: Convolu- tional neural networks in eeg analysis,
Robin Schirrmeister, Jost Springenberg, Lukas Fiederer, Martin Glasstetter, Katharina Eggensperger, Michael Tangermann, Frank Hutter, Wolfram Burgard, and To- nio Ball, “Deep learning with convolutional neural net- works for eeg decoding and visualization: Convolu- tional neur...
2017
-
[20]
Simple and scalable predictive un- certainty estimation using deep ensembles,
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell, “Simple and scalable predictive un- certainty estimation using deep ensembles,” Advances in neural information processing systems, vol. 30, 2017
2017
-
[21]
Overconfi- dence is key: Verbalized uncertainty evaluation in large language and vision-language models,
Tobias Groot and Matias Valdenegro-Toro, “Overconfi- dence is key: Verbalized uncertainty evaluation in large language and vision-language models,” arXiv preprint arXiv:2405.02917, 2024
2024 arXiv
-
[22]
Assessment and comparison of prog- nostic classification schemes for survival data,
Erika Graf, Claudia Schmoor, Willi Sauerbrei, and Mar- tin Schumacher, “Assessment and comparison of prog- nostic classification schemes for survival data,” Statis- tics in medicine , vol. 18, no. 17-18, pp. 2529–2545, 1999
1999
-
[23]
Bayesian opportunities for brain–computer interfaces: Enhancement of the existing classification algorithms and out-of-domain detection,
Egor I Chetkin, Sergei L Shishkin, and Bogdan L Kozyrskiy, “Bayesian opportunities for brain–computer interfaces: Enhancement of the existing classification algorithms and out-of-domain detection,” Algorithms, vol. 16, no. 9, pp. 429, 2023
2023
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.