REVIEW 5 major objections 5 minor 20 references
Uncertainty Quantification in the Tsetlin Machine
T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A linear formula turns any Tsetlin machine's class sum into a probability score.
desk verdict A useful heuristic for TM uncertainty, but the central calibration claim collapses when negative-polarity clauses are included. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the probability score $P(y=1|x)=\frac{1}{2}(1+v(x)/T)$, where $v(x)$ is the weighted class sum (matching positive clauses minus matching negative clauses) and $T$ is the target value that clips the class sum. The derivation mechanism is the equilibrium condition $P(I|x)=P(II|x)$: setting the probabilities of a clause receiving Type I versus Type II feedback equal, then solving for $v(x)$, yields $v=T[2P(y=1|x)-1]$, which inverts to the probability score. Since the class sum is already clipped to $[-T,T]$, the same map is also the most natural way to rescale the class sum to a probability. This machinery carries the argument by turning a learning-dynamics balance into an inference-time probability estimate.
What would settle it
Run the Section 3.1 single-pattern noise experiment and record, per epoch, the empirical frequencies with which the pattern receives Type I and Type II feedback; if those frequencies do not approach equality even once the class sum has reached its stationary oscillation, the balance condition behind Eq. (7) is violated and the probability score's theoretical grounding fails.
Extended reading notes
Core claim
The central discovery is that the class sum is not just a voting tally but a probability surrogate. The paper derives $P(y=1|x)=\frac{1}{2}(1+v(x)/T)$ by setting the total probability of Type I feedback for a positive clause equal to the total probability of Type II feedback for that clause, then solving for $v(x)$. The same formula is the natural affine rescaling of the clipped class sum to the unit interval, so it also holds as a scaling identity even without the equilibrium argument. The authors claim the score is inherent to any Tsetlin machine variant, applies to multiclass problems after normalizing by the sum of per-class scores, and behaves sensibly outside the training domain, producing near-0.5 scores instead of overconfident extrapolation.
Load-bearing premise
The argument assumes that during training the chance of a clause receiving reinforcing feedback and the chance of it receiving correcting feedback become equal at equilibrium; the paper asserts this balance rather than proving it from the automaton update rules, and the plotted class sums keep oscillating instead of settling at a fixed value.
Editorial extensions
If this is right
- Any trained Tsetlin machine can emit a probability score for each prediction from its class sum and $T$ alone; no calibration set or temperature scaling is needed.
- The score provides an abstention rule: with a chosen threshold, only predictions above it are accepted. On the CIFAR-10 model, thresholding at normalized score 0.6 raises accuracy from 65% to 93.2% on the 1,231 accepted samples.
- Outside the training-data region, binary-class scores fall back toward 0.5, so the model signals low confidence in extrapolation rather than overconfident predictions as neural networks often do.
- For multiclass problems, the raw per-class score overstates certainty because it reflects one-vs-rest confidence; the paper's normalized score, the largest class score divided by the sum of all class scores, is the more honest certainty measure.
- Because the score needs no true labels, it can be averaged over several epochs during training to smooth the oscillation of class sums, at the cost of slower inference.
Reading between the lines
- A natural next test, not performed in the paper, is to benchmark the probability score as an out-of-distribution detector: the near-0.5 score outside the training domain suggests that low-scoring or near-0.5 samples can flag inputs far from the training data.
- The alternative derivation as a pure affine rescaling means Eq. (7) can be viewed as an implicit calibration layer; one could combine it with distribution-free post-processing to produce coverage guarantees for selective prediction, which the paper does not explore.
- Because the score is a deterministic function of the class sum, it should transfer to any Tsetlin machine variant that maintains a class sum, but the paper asserts this without a separate calibration experiment for each variant; a cross-variant study would settle the claim.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a probability score for Tsetlin Machine (TM) predictions, P(y=1|x) = 1/2 (1 + v(x)/T), derived by balancing Type I and Type II feedback probabilities in Section 2.2. It claims this score is an inherent property of any TM variant. Experiments on simulated data show class sums roughly track noise levels, a two-feature moons visualization shows confidence decreasing outside the training domain, and a CIFAR-10 study uses a normalized multiclass score to show that thresholding on the score can improve accuracy on selected subsets. The paper concludes with limitations, including the acknowledged oscillation of class sums.
Significance. If the central derivation were sound, the paper would contribute a parameter-free calibration map from TM class sums to probabilities, with practical value for selective prediction and interpretability. The paper also ships a falsifiable empirical pattern -- probabilities outside the training domain approach 0.5 -- which is interesting if reproducible. However, the significance is conditional on the central claim, which is not established in the manuscript. The paper is honest about the oscillation limitation and the heuristic nature of the multiclass normalization, which narrows what is being claimed.
major comments (5)
- [Section 2.2, Eqs. (4)-(6)] The derivation of Eq. (6) balances P(I|x) and P(II|x) for a positive-polarity clause using P(I|x) = P_I(x) P(y=1|x) and P(II|x) = P_II(x) P(y=0|x). The text notes that for a negative-polarity clause the two probabilities are opposite, but it never uses this fact. For a negative clause, the balance condition is P(I|x) = P_I(x) P(y=0|x) and P(II|x) = P_II(x) P(y=1|x), which yields the opposite equilibrium v = -T[2P(y=1|x)-1]. Since v(x) is a single class sum, both conditions can hold only in the degenerate case v=0 and P(y=1|x)=1/2. Thus Eq. (7) cannot be the equilibrium of a two-polarity TM under the paper's own feedback model, and the claim that it is 'derived through an analysis of the TM learning dynamics' is not supported.
- [Section 4, Limitations; Figure 1] The paper itself acknowledges that the class sum 'oscillates and changes from epoch to epoch' rather than converging to a fixed point. That acknowledged behavior is in tension with the equilibrium derivation of Eq. (6), which assumes a stationary balance of feedback probabilities. The paper does not show that the time-averaged class sum is the fixed point of the clause-update dynamics, so the connection between the oscillation average and Eq. (7) is asserted rather than demonstrated.
- [Section 2.3, Eq. (7)] The alternative derivation in Section 2.3 is purely a rescaling and shifting of the clipped class sum to [0,1]. This confirms that Eq. (7) is the natural normalization of v(x), but it does not establish that the resulting value is a calibrated probability. The claim that the probability score is 'coincidentally the same probability as receiving Type II feedback' is also not meaningful for calibration unless the feedback balance is proven. Without independent evidence, Eq. (7) remains a heuristic score, not a derived probability.
- [Section 3.1, Figure 1; Section 3.2.2, Figures 5-7] The main empirical support for the probability-score interpretation is the statement that 'when using Eq. (7) on the average of the last 200 epochs, the resulting probability scores are approximately the same as the noise levels (not shown here).' This is a central quantitative claim, but no plot, table, or numeric error is given, and no error bars are provided for the single-pattern or CIFAR experiments. Similarly, Figure 7 reports accuracies for thresholded subsets without confidence intervals; with only 10,000 test samples and 65% base accuracy, the 0.932 figure needs a standard error or bootstrap interval to be assessable.
- [Section 3.2.2, normalized score definition] The proposed multiclass normalized score (max individual probability divided by the sum of the ten individual probability scores) is introduced without a derivation or justification beyond a qualitative rationale. Its validity as a certainty measure is not connected to the learning dynamics, and no comparison with alternatives (e.g., softmax of the class sums, margin score, or entropy) is given. Since the CIFAR-10 conclusions rely entirely on this score, the paper's main application result is not fully supported.
minor comments (5)
- [Section 3.1.1, Figure 3] The color scale for Figure 3 is not described in the text, and the overlap region and the outside-domain regions are identified by eye; a contour line at 0.5 or a quantitative comparison of the TM and MLP extrapolation behavior would make the claimed contrast precise.
- [Section 3.2.2, CIFAR-10 setup] The hyperparameters for the two s-values (s=20 and s=1.2) are given in the text, but it is not stated how many runs were performed or whether the reported 65% and 63% accuracies are single runs; state the number of seeds or runs.
- [Section 2.1, Eq. (3)] The weighted class sum is introduced but the weight update rule is never specified. Since the subsequent theory uses v(x) in the feedback formulas as if the weights were subsumed, a sentence clarifying the relationship between weights and the clipping to [-T,T] would be helpful.
- [Throughout] Typographical and grammatical issues include 'tends to decrease the class sum' (singular/plural agreement), 'the the accuracy', 'with with different degree', and the incomplete sentence in the abstract's surrounding text; a careful proofread is needed.
- [References] The reference list misses recent uncertainty-aware TM work that could provide a comparison baseline, and the paper would benefit from citing the formal convergence results of Jiao et al. and Zhang et al. in the derivation section, since those results concern feedback dynamics that are directly relevant to the equilibrium assumption.
Circularity Check
The probability score of Eq. (7) is the affine normalization of the clipped class sum; the 'learning-dynamics derivation' re-inserts the same affine function through the feedback probability PII and ignores negative-polarity clauses.
-
self definitional
[Section 2.1-2.2, Eq. (7) and the feedback probability PII]
"If the sample does not belong to the target class, the clause is given reject feedback with probability PII(xi)=1/2(1+v(xi)/T). ... From this equation, one find that the probability P(y=1|x) for the target to be true given a sample can be estimated by the equation: P(y=1|x)=1/2(1+v(xi)/T), which is coincidentally the same probability as receiving Type II feedback."
PII is introduced as 1/2(1+v/T), and Eq. (7) is literally the same function of v. Since PI=1-PII, the equilibrium condition P(I|x)=P(II|x) reduces algebraically to (1-PII)P(y=1|x)=PII(1-P(y=1|x)), which forces P(y=1|x)=PII. Thus Eq. (7) is not an outcome of independent dynamics but a rearrangement of the feedback probability that was already defined as an affine function of the class sum; no new class-sum-to-probability information is added by the derivation.
-
renaming known result
[Section 2.3, Alternative derivation of probability score]
"Since the class sum is clipped at -T and T, you first scale the class sum by T to get values in the range -1 and 1. Then, you shift the score to the range 0 and 2 by adding 1. Finally, you divide that by 2 to get the range of 0 and 1 needed for a valid probability score. Putting all these steps together, you get the equation 1/2(1+v(x)/T), which is the exact same equation as Eq. (7) which was derived by setting TypeI and TypeII feedback probabilities equal each other."
This passage explicitly states that Eq. (7) is fully determined by the clipping bounds of the class sum: any bounded score can be affinely rescaled to [0,1]. Consequently, the 'probability score' is a normalization of v introduced by construction; calling it a probability is a labeling choice rather than an empirical finding. The earlier feedback-balance derivation is therefore redundant for identifying the functional form, and the abstract's claim that the score is 'derived through an analysis of the TM learning dynamics' overstates what the equations establish.
full rationale
The two steps above identify a real but partial circularity. The empirical sections are not circular: Figure 1 tests whether learned class sums track known noise probabilities, and the CIFAR-10 threshold analysis is an external benchmark whose accuracy-vs-confidence result would stand or fall independently of how Eq. (7) is labelled. The paper is also transparent in Section 2.3 that the formula is the natural scaling of a clipped sum. Nevertheless, the central theoretical contribution reduces by construction: Eq. (7) is identical to the definition of PII and to the affine rescaling of v, so its functional form is not independently derived from the TM learning dynamics. The derivation in Section 2.2 further assumes a feedback balance ('there is a point where the probabilities of the two feedback types ... is the same') without deriving it, applies only to positive-polarity clauses, and is contradicted by the paper's own Figure 1 and Section 4 admission that class sums oscillate rather than settle; for negative-polarity clauses the mirror balance gives the opposite equilibrium, so the claimed joint equilibrium cannot exist in a two-polarity TM. These are correctness flaws rather than self-citation or fitted-input circularity, but they leave the 'derived' status of Eq. (7) as mostly definitional, with genuine empirical support confined to the class-sum tracking experiments and the external CIFAR-10 threshold evaluation.
Assumptions & free parameters
free parameters (2)
- T (target value) =
T=2000, 10,000, 20,000 in experiments
- s (specificity) =
s=1.0, 1.1, 1.2, 2.0, 5.0, 20 in experiments
assumptions (3)
- ad hoc to paper At equilibrium, the probability of Type I feedback equals the probability of Type II feedback for a clause: P(I|x)=P(II|x).
- domain assumption The class sum is clipped to [-T,T] for every sample.
- ad hoc to paper The normalized multiclass score, defined as the largest individual probability score divided by the sum of scores over all classes, is a valid certainty measure.
invented entities (1)
-
Normalized probability score (max score divided by sum of scores)
Cite this review
Pith. "Pith review of Uncertainty Quantification in the Tsetlin Machine." pith.science (2026). https://pith.science/paper/5CYT4WXD
@misc{pith2026250704175,
author = {Pith},
title = {Pith review of: Uncertainty Quantification in the Tsetlin Machine},
year = {2026},
howpublished = {\url{https://pith.science/paper/5CYT4WXD}},
note = {Machine review of arXiv:2507.04175}
}
read the original abstract
Data modeling using Tsetlin machines (TMs) is all about building logical rules from the data features. The decisions of the model are based on a combination of these logical rules. Hence, the model is fully transparent and it is possible to get explanations of its predictions. In this paper, we present a probability score for TM predictions and develop new techniques for uncertainty quantification to increase the explainability further. The probability score is an inherent property of any TM variant and is derived through an analysis of the TM learning dynamics. Simulated data is used to show a clear connection between the learned TM probability scores and the underlying probabilities of the data. A visualization of the probability scores also reveals that the TM is less confident in its predictions outside the training data domain, which contrasts the typical extrapolation phenomenon found in Artificial Neural Networks. The paper concludes with an application of the uncertainty quantification techniques on an image classification task using the CIFAR-10 dataset, where they provide new insights and suggest possible improvements to current TM image classification models.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
Darshana Abeyrathna, Ole-Christoffer Granmo, Lei Jiao, and Morten Goodwin
K. Darshana Abeyrathna, Ole-Christoffer Granmo, Lei Jiao, and Morten Goodwin. The regression tsetlin machine: A tsetlin machine for continuous output problems. In Paulo Moura Oliveira, Paulo Novais, and Lu \'i s Paulo Reis, editors, Progress in Artificial Intelligence, pages 268--280, Cham, 2019. Springer International Publishing
work page 2019
-
[2]
K. Darshana Abeyrathna, Ahmed A. O. Abouzeid, Bimal Bhattarai, Charul Giri, Sondre Glimsdal, Ole-Christoffer Granmo, Lei Jiao, Rupsa Saha, Jivitesh Sharma, Svein A. Tunheim, and Xuan Zhang. Building concise logical patterns by constraining Tsetlin Machine clause size . In IJCAI, 2023
work page 2023
-
[3]
Kuruge Darshana Abeyrathna, Bimal Bhattarai, Morten Goodwin, Saeed Rahimi Gorji, Ole-Christoffer Granmo, Lei Jiao, Rupsa Saha, and Rohan K Yadav. Massively parallel and asynchronous Tsetlin Machine architecture supporting almost constant-time scaling . In ICML, 2021
work page 2021
-
[4]
Tsetlin Machine embedding: Representing words using logical expressions
Bimal Bhattarai, Ole-Christoffer Granmo, Lei Jiao, Rohan Yadav, and Jivitesh Sharma. Tsetlin Machine embedding: Representing words using logical expressions . Findings of EACL, pages 1512--1522, 2024
work page 2024
-
[5]
Ole-Christoffer Granmo. The tsetlin machine--a game theoretic bandit driven approach to optimal pattern recognition with propositional logic. arXiv preprint arXiv:1804.01508, 2018
arXiv 2018
-
[6]
The convolutional tsetlin machine
Ole-Christoffer Granmo, Sondre Glimsdal, Lei Jiao, Morten Goodwin, Christian W Omlin, and Geir Thore Berge. The convolutional tsetlin machine. arXiv preprint arXiv:1905.09688, 2019
arXiv 1905
-
[7]
Smørvik, and Ole-Christoffer Granmo
Ylva Grønningsæter, Halvor S. Smørvik, and Ole-Christoffer Granmo. An optimized toolbox for advanced image processing with tsetlin machine composites. In 2024 International Symposium on the Tsetlin Machine (ISTM), pages 1--8, 2024. doi:10.1109/ISTM62799.2024.10931429
-
[8]
Interpretable rule-based architecture for GNSS jamming signal classification
Sindhusha Jeeru, Lei Jiao, Per-Arne Andersen, and Ole-Christoffer Granmo. Interpretable rule-based architecture for GNSS jamming signal classification. IEEE Sensors Journal, 2025
work page 2025
Show all 20 references
-
[9]
Darshana Abeyrathna
Lei Jiao, Xuan Zhang, Ole-Christoffer Granmo, and K. Darshana Abeyrathna . On the convergence of Tsetlin machines for the XOR operator . IEEE Trans. Pattern Anal. Mach. Intell., 45 0 (5): 0 6072--6085, Jan. 2023
2023
-
[10]
Learning multiple layers of features from tiny images
Alex Krizhevsky. Learning multiple layers of features from tiny images. University of Toronto, 2009. URL https://api.semanticscholar.org/CorpusID:18268744
2009
-
[11]
REDRESS: Generating compressed models for edge inference using Tsetlin Machines
Sidharth Maheshwari, Tousif Rahman, Rishad Shafik, Alex Yakovlev, Ashur Rafiev, Lei Jiao, and Ole-Christoffer Granmo. REDRESS: Generating compressed models for edge inference using Tsetlin Machines . IEEE Transactions on Pattern Analysis and Machine Intelligence, 45 0 (9): 0 1...
2023
-
[12]
Pedregosa, G
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in P ython. Journal of Machine Learnin...
2011
-
[13]
FedTMOS : Efficient one-shot federated learning with T setlin machine
Shannon How Shi Qi, Jagmohan Chauhan, Geoff V Merrett, and Jonathon Hare. FedTMOS : Efficient one-shot federated learning with T setlin machine. In The Thirteenth International Conference on Learning Representations (ICLR), 2025
2025
-
[14]
Zadorozhny, and Morten Goodwin
Rupsa Saha, Ole-Christoffer Granmo, Vladimir I. Zadorozhny, and Morten Goodwin. A relational tsetlin machine with applications to natural language understanding. Journal of intelligent information systems, 59 0 (1): 0 121--148, 2022
2022
-
[15]
Tsetlin Machine for solving contextual bandit problems
Raihan Seraj, Jivitesh Sharma, and Ole-Christoffer Granmo. Tsetlin Machine for solving contextual bandit problems . In NeurIPS, 2022
2022
-
[16]
Drop clause: Enhancing performance, robustness and pattern recognition capabilities of the Tsetlin Machine
Jivitesh Sharma, Rohan Yadav, Ole-Christoffer Granmo, and Lei Jiao. Drop clause: Enhancing performance, robustness and pattern recognition capabilities of the Tsetlin Machine . In AAAI, 2023
2023
-
[17]
Tsetlin machine-based image classification FPGA accelerator with on-device training
Svein Anders Tunheim, Lei Jiao, Rishad Shafik, Alex Yakovlev, and Ole-Christoffer Granmo. Tsetlin machine-based image classification FPGA accelerator with on-device training. IEEE Trans. Circuits Syst. I, Reg. Papers, 72 0 (2): 0 830--843, Feb. 2025
2025
-
[18]
Human-level interpretable learning for aspect-based sentiment analysis
Rohan K Yadav, Lei Jiao, Ole-Christoffer Granmo, and Morten Goodwin. Human-level interpretable learning for aspect-based sentiment analysis . In AAAI, 2021
2021
-
[19]
Robust interpretable text classification against spurious correlations using and-rules with negation
Rohan Kumar Yadav, Jiao Lei, Ole-Christoffer Granmo, and Morten Goodwin. Robust interpretable text classification against spurious correlations using and-rules with negation. In Proc. 31st Int. Joint Conf. Artif. Intell. (IJCAI). International Joint Conferences on Artificial I...
2022
-
[20]
On the convergence of Tsetlin machines for the IDENTITY - and NOT Operators
Xuan Zhang, Lei Jiao, Ole-Christoffer Granmo, and Morten Goodwin. On the convergence of Tsetlin machines for the IDENTITY - and NOT Operators . IEEE Trans. Pattern Anal. Mach. Intell., 44 0 (10): 0 6345--6359, Jul. 2022
2022
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.