{"id":"b36a40ad-1256-4948-af25-4cdac0372f3e","arxiv_id":"2507.04175","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A Tsetlin machine's class sum is mapped to a probability score by P=(1+v/T)/2 and used for uncertainty quantification, but the derivation is heuristic and the validation is weak.","lead":"This paper turns the internal vote count of a Tsetlin machine into a probability-like score for how sure the model is. It applies the score to simulated data and CIFAR-10 images, but the derivation and validation are not yet strong enough to call it calibrated.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (7)'s derivation balances feedback for positive clauses only; the analogous balance for negative clauses gives the opposite condition, so the claimed class-sum equilibrium cannot exist in a two-polarity TM.","rationale":"The most load-bearing concern about the central claim is not the empirical shortfalls (which the reader documents: omitted quantitative comparison, no ECE, no code), but the internal invalidity of the derivation itself. The central claim is that Eq. (7) is an inherent property of any TM variant derived from learning dynamics. To support that, the paper must show the class-sum balance is a genuine fixed point of the TM's feedback process. The derivation only balances feedback for positive clauses and ignores the opposite-polarity clauses that are part of every standard TM. When the negative-clause balance or the total drift is computed, Eq. (6) is not the fixed point for any realistic symmetric TM. This is a stronger failure than 'the balance is not proven': even if one grants the paper's qualitative account of feedback effects, the proposed equilibrium is the wrong one and cannot be reached. The empirical claims ('not shown here') and the rescaling in §2.3 cannot repair this. The reader's verdict of REJECT remains appropriate, so no verdict change is needed.","tokens_in":9516,"tokens_out":14714,"duration_ms":153231,"concrete_test":"Analytically re-derive the class-sum equilibrium by writing the expected one-step change in v(x) under the paper's own definitions: E[Δv] = Σ_{C+}(P(I|x)-P(II|x)) - Σ_{C-}(P(II|x)-P(I|x)), then solve E[Δv]=0. For n+=n-, the solution is p=1/2 independent of v; for n+≠n-, v = T(n+ + n-)/(n+ - n-)(2p-1). If the resulting fixed point differs from Eq. (6) for the symmetric clause counts used in §3.1, the balance condition in §2.2 is not a valid equilibrium and Eq. (7) has no theoretical grounding.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central probability claim is Eq. (7), P(y=1|x)=1/2(1+v/T), derived in §2.2 by setting P(I|x)=P(II|x) from Eqs. (4)-(5). Those equations are stated for a positive-polarity clause. The paper notes that for a negative-polarity clause 'the two probabilities are opposite' but does not use this. For a negative clause, the balance condition is P(II|x)=PII(x)P(y=1|x) and P(I|x)=PI(x)P(y=0|x), and setting them equal gives v = -T[2P(y=1|x)-1]. Since v is a single global class sum, the two polarities require opposite equilibria; both can hold only in the degenerate case v=0, p=1/2. Under the paper's own sign convention (Type I increases v, Type II decreases it, reversed for negative clauses), the expected one-step drift of v is E[Δv] = n+(PI p - PII(1-p)) + n-(PII p - PI(1-p)) = (n+ + n-)(2p-1) + (n- - n+)v/T. For the standard symmetric TM, n+=n-, this drift is nonzero whenever p≠1/2, independent of v, so Eq. (6) is not the fixed point of the class-sum dynamics. The oscillations reported in Figure 1 and acknowledged in Section 4 are consistent with this missing balance. Section 2.3's 'alternative derivation' is only a rescaling of the clipped v and does not establish calibration. Thus Eq. (7) is not 'derived through an analysis of the TM learning dynamics' as claimed; it is an unsupported heuristic.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a probability score for Tsetlin Machine (TM) predictions, P(y=1|x) = 1/2 (1 + v(x)/T), derived by balancing Type I and Type II feedback probabilities in Section 2.2. It claims this score is an inherent property of any TM variant. Experiments on simulated data show class sums roughly track noise levels, a two-feature moons visualization shows confidence decreasing outside the training domain, and a CIFAR-10 study uses a normalized multiclass score to show that thresholding on the score can improve accuracy on selected subsets. The paper concludes with limitations, including the acknowledged oscillation of class sums.","tokens_in":9868,"tokens_out":4839,"duration_ms":33431,"significance":"If the central derivation were sound, the paper would contribute a parameter-free calibration map from TM class sums to probabilities, with practical value for selective prediction and interpretability. The paper also ships a falsifiable empirical pattern -- probabilities outside the training domain approach 0.5 -- which is interesting if reproducible. However, the significance is conditional on the central claim, which is not established in the manuscript. The paper is honest about the oscillation limitation and the heuristic nature of the multiclass normalization, which narrows what is being claimed.","major_comments":[{"comment":"The derivation of Eq. (6) balances P(I|x) and P(II|x) for a positive-polarity clause using P(I|x) = P_I(x) P(y=1|x) and P(II|x) = P_II(x) P(y=0|x). The text notes that for a negative-polarity clause the two probabilities are opposite, but it never uses this fact. For a negative clause, the balance condition is P(I|x) = P_I(x) P(y=0|x) and P(II|x) = P_II(x) P(y=1|x), which yields the opposite equilibrium v = -T[2P(y=1|x)-1]. Since v(x) is a single class sum, both conditions can hold only in the degenerate case v=0 and P(y=1|x)=1/2. Thus Eq. (7) cannot be the equilibrium of a two-polarity TM under the paper's own feedback model, and the claim that it is 'derived through an analysis of the TM learning dynamics' is not supported.","section":"Section 2.2, Eqs. (4)-(6)"},{"comment":"The paper itself acknowledges that the class sum 'oscillates and changes from epoch to epoch' rather than converging to a fixed point. That acknowledged behavior is in tension with the equilibrium derivation of Eq. (6), which assumes a stationary balance of feedback probabilities. The paper does not show that the time-averaged class sum is the fixed point of the clause-update dynamics, so the connection between the oscillation average and Eq. (7) is asserted rather than demonstrated.","section":"Section 4, Limitations; Figure 1"},{"comment":"The alternative derivation in Section 2.3 is purely a rescaling and shifting of the clipped class sum to [0,1]. This confirms that Eq. (7) is the natural normalization of v(x), but it does not establish that the resulting value is a calibrated probability. The claim that the probability score is 'coincidentally the same probability as receiving Type II feedback' is also not meaningful for calibration unless the feedback balance is proven. Without independent evidence, Eq. (7) remains a heuristic score, not a derived probability.","section":"Section 2.3, Eq. (7)"},{"comment":"The main empirical support for the probability-score interpretation is the statement that 'when using Eq. (7) on the average of the last 200 epochs, the resulting probability scores are approximately the same as the noise levels (not shown here).' This is a central quantitative claim, but no plot, table, or numeric error is given, and no error bars are provided for the single-pattern or CIFAR experiments. Similarly, Figure 7 reports accuracies for thresholded subsets without confidence intervals; with only 10,000 test samples and 65% base accuracy, the 0.932 figure needs a standard error or bootstrap interval to be assessable.","section":"Section 3.1, Figure 1; Section 3.2.2, Figures 5-7"},{"comment":"The proposed multiclass normalized score (max individual probability divided by the sum of the ten individual probability scores) is introduced without a derivation or justification beyond a qualitative rationale. Its validity as a certainty measure is not connected to the learning dynamics, and no comparison with alternatives (e.g., softmax of the class sums, margin score, or entropy) is given. Since the CIFAR-10 conclusions rely entirely on this score, the paper's main application result is not fully supported.","section":"Section 3.2.2, normalized score definition"}],"minor_comments":[{"comment":"The color scale for Figure 3 is not described in the text, and the overlap region and the outside-domain regions are identified by eye; a contour line at 0.5 or a quantitative comparison of the TM and MLP extrapolation behavior would make the claimed contrast precise.","section":"Section 3.1.1, Figure 3"},{"comment":"The hyperparameters for the two s-values (s=20 and s=1.2) are given in the text, but it is not stated how many runs were performed or whether the reported 65% and 63% accuracies are single runs; state the number of seeds or runs.","section":"Section 3.2.2, CIFAR-10 setup"},{"comment":"The weighted class sum is introduced but the weight update rule is never specified. Since the subsequent theory uses v(x) in the feedback formulas as if the weights were subsumed, a sentence clarifying the relationship between weights and the clipping to [-T,T] would be helpful.","section":"Section 2.1, Eq. (3)"},{"comment":"Typographical and grammatical issues include 'tends to decrease the class sum' (singular/plural agreement), 'the the accuracy', 'with with different degree', and the incomplete sentence in the abstract's surrounding text; a careful proofread is needed.","section":"Throughout"},{"comment":"The reference list misses recent uncertainty-aware TM work that could provide a comparison baseline, and the paper would benefit from citing the formal convergence results of Jiao et al. and Zhang et al. in the derivation section, since those results concern feedback dynamics that are directly relevant to the equilibrium assumption.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is borderline between major revision and reject. The skeptical argument in the stress-test note (opposite polarity balance gives the opposite equilibrium) appears to be correct on its face and is not addressed in the manuscript. I therefore do not recommend reject only because the paper's empirical scoring pipeline could be salvaged as a heuristic if the calibration claims were replaced by a more modest descriptive claim about class sums. I would ask the editor to check whether the authors' prior work or the broader TM literature already contains convergence results for multiclass TMs that would resolve the polarity issue. The missing error bars and the 'not shown here' quantitative claim are also concerning under the journal's reproducibility standards."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about 2507.04175. First, it proposes a clean, simple way to turn Tsetlin machine class sums into a [0,1] score: P = 0.5(1 + v/T). That explicit formula is new for TMs. Second, the paper's claim that this is a calibrated probability \"derived through an analysis of the TM learning dynamics\" does not hold up. The derivation in Sec. 2.2 balances Type I and Type II feedback only for positive-polarity clauses. For negative-polarity clauses, the balance condition is v = -T(2P(y=1|x)-1), the opposite sign. Since v is a single global class sum, both equilibria cannot exist except in the degenerate case. A drift calculation confirms this: with symmetric clause counts, the expected change in v is proportional to 2p-1, independent of v, so Eq. (6) is not a fixed point. The paper's own Fig. 1 and Sec. 4 show class sums oscillating, which is consistent with the missing balance.\n\nWhat is genuinely useful: the linear score is a natural normalization of the clipped class sum, and the paper says so explicitly in Sec. 2.3. The simulations show a monotone relationship between average class sum and underlying noise, which is intuitive; the CIFAR-10 threshold experiment shows that filtering by the normalized score boosts accuracy from 65% to 93% on the most confident 12% of samples. That is a reasonable heuristic for selective prediction.\n\nThe soft spots are the central derivation, plus the empirical gaps: the key quantitative claim (that scores match noise levels) is literally \"not shown here\"; there are no error bars, no baselines, no calibration metrics like ECE or Brier, and no code. The CIFAR-10 normalized score (max score divided by sum) is introduced ad hoc and is not derived. None of these are fatal for a heuristic, but they are fatal for the abstract's assertion that the score is an inherent calibrated property.\n\nMy take: the paper is a modest extension with a useful heuristic, but it overclaims. The equilibrium derivation should be dropped or corrected, and the empirical section needs error bars, calibration curves, and released code.\n\nRecommendation: send to peer review. The idea is worth a serious round of review, and the authors have been honest about limitations; with a major revision this could become a decent contribution to the TM literature.","headline":"A useful heuristic for TM uncertainty, but the central calibration claim collapses when negative-polarity clauses are included.","tokens_in":10448,"tokens_out":5586,"would_cite":false,"duration_ms":60070,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A linear formula turns any Tsetlin machine's class sum into a probability score.","keywords":["Tsetlin machine","uncertainty quantification","probability score","class sum","selective prediction","explainability","CIFAR-10","feedback equilibrium"],"falsifier":"Run the Section 3.1 single-pattern noise experiment and record, per epoch, the empirical frequencies with which the pattern receives Type I and Type II feedback; if those frequencies do not approach equality even once the class sum has reached its stationary oscillation, the balance condition behind Eq. (7) is violated and the probability score's theoretical grounding fails.","tokens_in":9271,"feed_emoji":"📊","tokens_out":8270,"duration_ms":83212,"temperature":0.7,"pith_summary":"Tsetlin machines classify by counting logical clauses that vote for and against a target, and the difference of those counts is the class sum $v(x)$, clipped to $[-T,T]$. This paper claims that $v(x)$ carries calibrated probability information: at learning equilibrium, the probability that a clause receives reinforcing feedback equals the probability it receives correcting feedback, and solving that balance gives $P(y=1|x)=\\frac{1}{2}(1+v(x)/T)$. If correct, any Tsetlin machine already outputs a probability score at inference time without retraining or extra calibration. The paper tests this on simulated data and on CIFAR-10 image classification, where thresholding the normalized score at 0.6 yields 93.2% accuracy on 1,231 high-certainty test samples. The practical payoff is a built-in certainty signal for a transparent rule-based model, enabling selective prediction and more human-readable explanations.","feed_headline":"Tsetlin machine class sums are calibrated probabilities","feed_subtitle":"One linear formula, no retraining: threshold it to abstain and lift CIFAR-10 accuracy from 65% to 93.2%.","key_machinery":"The central object is the probability score $P(y=1|x)=\\frac{1}{2}(1+v(x)/T)$, where $v(x)$ is the weighted class sum (matching positive clauses minus matching negative clauses) and $T$ is the target value that clips the class sum. The derivation mechanism is the equilibrium condition $P(I|x)=P(II|x)$: setting the probabilities of a clause receiving Type I versus Type II feedback equal, then solving for $v(x)$, yields $v=T[2P(y=1|x)-1]$, which inverts to the probability score. Since the class sum is already clipped to $[-T,T]$, the same map is also the most natural way to rescale the class sum to a probability. This machinery carries the argument by turning a learning-dynamics balance into an inference-time probability estimate.","core_discovery":"The central discovery is that the class sum is not just a voting tally but a probability surrogate. The paper derives $P(y=1|x)=\\frac{1}{2}(1+v(x)/T)$ by setting the total probability of Type I feedback for a positive clause equal to the total probability of Type II feedback for that clause, then solving for $v(x)$. The same formula is the natural affine rescaling of the clipped class sum to the unit interval, so it also holds as a scaling identity even without the equilibrium argument. The authors claim the score is inherent to any Tsetlin machine variant, applies to multiclass problems after normalizing by the sum of per-class scores, and behaves sensibly outside the training domain, producing near-0.5 scores instead of overconfident extrapolation.","pith_inferences":["A natural next test, not performed in the paper, is to benchmark the probability score as an out-of-distribution detector: the near-0.5 score outside the training domain suggests that low-scoring or near-0.5 samples can flag inputs far from the training data.","The alternative derivation as a pure affine rescaling means Eq. (7) can be viewed as an implicit calibration layer; one could combine it with distribution-free post-processing to produce coverage guarantees for selective prediction, which the paper does not explore.","Because the score is a deterministic function of the class sum, it should transfer to any Tsetlin machine variant that maintains a class sum, but the paper asserts this without a separate calibration experiment for each variant; a cross-variant study would settle the claim."],"forward_implications":["Any trained Tsetlin machine can emit a probability score for each prediction from its class sum and $T$ alone; no calibration set or temperature scaling is needed.","The score provides an abstention rule: with a chosen threshold, only predictions above it are accepted. On the CIFAR-10 model, thresholding at normalized score 0.6 raises accuracy from 65% to 93.2% on the 1,231 accepted samples.","Outside the training-data region, binary-class scores fall back toward 0.5, so the model signals low confidence in extrapolation rather than overconfident predictions as neural networks often do.","For multiclass problems, the raw per-class score overstates certainty because it reflects one-vs-rest confidence; the paper's normalized score, the largest class score divided by the sum of all class scores, is the more honest certainty measure.","Because the score needs no true labels, it can be averaged over several epochs during training to smooth the oscillation of class sums, at the cost of slower inference."],"supporting_citations":[{"why":"Defines the Tsetlin machine learning dynamics, feedback types, and the class-sum and target-value parameters that Eq. (7) builds on.","marker":"Granmo [2018]"},{"why":"Supplies the literal-budget constraint used to train the CIFAR-10 model whose class sums are analyzed.","marker":"Abeyrathna et al. [2023]"},{"why":"Provides the composite Tsetlin machine accuracy baseline and the thermometer encoding setup for the CIFAR-10 experiments.","marker":"Grønningsæter et al. [2024]"},{"why":"Supplies the CIFAR-10 dataset used in the image classification uncertainty application.","marker":"Krizhevsky [2009]"},{"why":"Supplies the make_moons simulated dataset used to visualize uncertainty regions and extrapolation behavior.","marker":"Pedregosa et al. [2011]"}],"fun_headline_variants":["Tsetlin class sums hide a calibrated probability score","No retraining: turn Tsetlin votes into reliable probabilities","Tsetlin uncertainty: calm extrapolation, better CIFAR-10 with abstention","A single affine formula makes Tsetlin scores probabilities"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes that during training the chance of a clause receiving reinforcing feedback and the chance of it receiving correcting feedback become equal at equilibrium; the paper asserts this balance rather than proving it from the automaton update rules, and the plotted class sums keep oscillating instead of settling at a fixed value.","fun_headline_variants_meta":{"raw":{"variants":["Tsetlin class sums hide a calibrated probability score","No retraining: turn Tsetlin votes into reliable probabilities","Tsetlin uncertainty: calm extrapolation, better CIFAR-10 with abstention","A single affine formula makes Tsetlin scores probabilities"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000282,"raw_usage":{"total_tokens":1637,"prompt_tokens":882,"completion_tokens":755,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":682}},"tokens_in":498,"tokens_out":755,"duration_ms":7938,"temperature":1.0,"reasoning_tokens":682,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:54:06.310163+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the Section 3.1 single-pattern noise experiment and record, per epoch, the empirical frequencies with which the pattern receives Type I and Type II feedback; if those frequencies do not approach equality even once the class sum has reached its stationary oscillation, the balance condition behind Eq. (7) is violated and the probability score's theoretical grounding fails.","supporting_citations":[{"cited_title":"Learning multiple layers of features from tiny images","cited_arxiv_id":null,"evidence_quote":"Supplies the CIFAR-10 dataset used in the image classification uncertainty application."}],"review_version":1}