REVIEW 3 major objections 6 minor 1 cited by
Kannada-MNIST: A new handwritten digits dataset for the Kannada language
T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Kannada-MNIST is a 60,000-image handwritten digit dataset in the Kannada script, designed as a drop-in MNIST replacement, whose worst-writer test set drops a standard CNN to 96.85% and an out-of-domain set to 76.2%.
desk verdict A genuinely useful Kannada digit dataset with open-sourced raw scans and code, but the 'drop-in replacement' framing overreaches because the headline accuracy is measured on a deliberately adversarial test cohort, not a random sample. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the worst-cohort split rule. Each of the 65 volunteers is scored by the mean probability that a MNIST-trained CNN classifies their Kannada 0 as 0 and their Kannada 3 and 7 as 2; the eight highest-scoring volunteers form the 10,000-image test set and the next 47 form the 60,000-image training set. This transforms the dataset from a within-writer recognition benchmark into a cross-writer generalization benchmark. The second object is Dig-MNIST, a 10,240-image out-of-domain test set written under different conditions by people new to the Kannada script, which provides a harder generalization target.
What would settle it
Train a CNN on 57 of the 65 volunteers and test on a randomly chosen eighth volunteer's digits instead of the worst-scoring cohort; if accuracy lands near the above-99% figure the paper saw in random splits, the 96.85% headline is an artifact of the selection rule. Inspecting per-writer accuracy inside the worst cohort would also show whether the difficulty is spread across all eight writers or driven by one or two.
Extended reading notes
Core claim
The paper's central claim is that a large, MNIST-compatible handwritten Kannada digit dataset can be built in the wild, and that the dominating challenge it poses is generalization to unseen writers, not recognition of familiar glyphs. Random splits between writers already reached above 99% accuracy, so the authors deliberately constructed the test set from the eight volunteers whose Kannada digits a MNIST-trained CNN found hardest to map to similar-looking MNIST classes; on that worst-cohort test set accuracy fell to 96.85%. On Dig-MNIST, collected with different writers, different paper, different scanner settings, and many first-time users of the script, the same model fell to 76.17%. Because the dataset, raw scans, and parsing code are all released, these numbers can be reproduced and the failure modes inspected.
Load-bearing premise
The load-bearing premise is that the worst-cohort selection rule—ranking volunteers by how often a MNIST-trained CNN maps their Kannada 0, 3, and 7 to the similar-looking MNIST classes 0 and 2—ranks true out-of-sample generalization difficulty, so the 96.85% figure describes what a classifier should expect on a new writer.
Editorial extensions
If this is right
- The reported 96.85% and 76.2% accuracies become reference baselines that any future Kannada digit classifier must beat.
- Dig-MNIST turns out-of-domain generalization into a measurable task: improve on 76.2% after training only on Kannada-MNIST, without image preprocessing, which the paper poses as an explicit open challenge.
- Because raw scans, scanner settings, and parsing code are released, researchers can test whether different segmentation pipelines change accuracy independently of the classifier.
- The quantitative comparisons—less peaked mean intensity, lower variance explained by the top PCA components, and different morphological trait distributions—give users a concrete sense of how Kannada-MNIST differs from MNIST.
- The released synthetic font seed code allows others to generate large synthetic Kannada digit sets, which the paper suggests as a route toward higher Dig-MNIST accuracy.
Reading between the lines
- Editorial inference: because random writer splits already give above 99% accuracy, the 96.85% headline should be read as a property of the worst-cohort selection rule, not of Kannada handwriting per se; the Dig-MNIST result is the cleaner measure of script-specific difficulty.
- Editorial inference: the Dig-MNIST drop may be partly caused by collection artifacts the paper documents—volunteers overrunning grid lines, smudges, and grid borders—so an ablation that removes partial strokes and border noise would separate segmentation artifacts from true out-of-domain glyph variation.
- Editorial inference: applying the same worst-cohort proxy to the original MNIST writers would provide a missing control, showing whether writer-split difficulty drops even for familiar Arabic numerals or is sharpened by Kannada's confusable 0, 3, and 7 glyphs.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Kannada-MNIST, a handwritten Kannada digits dataset of 60,000 training and 10,000 test images in MNIST-compatible 28x28 grayscale format, along with a separate 10,240-image Dig-MNIST out-of-domain test set. The authors document the data collection pipeline, including volunteer recruitment, scanning, grid segmentation, MNIST-ization, and a sanity check using a CNN pretrained on MNIST digits for the visually similar glyphs 0, 3, and 7. They provide morphological and dimensionality-reduction comparisons with MNIST and report CNN baselines: 96.85% accuracy on the main test set and 76.17% on Dig-MNIST. The central claim is that Kannada-MNIST can serve as a direct drop-in replacement for the original MNIST dataset and provides a sterner generalization challenge.
Significance. If the evaluation protocol is clarified, this is a valuable community resource: raw scans, scanner settings, preprocessing code, and model code are all open-sourced, which makes the dataset creation pipeline fully reproducible and supports end-to-end research on handwritten digit recognition in an Indic script. The paper also addresses a genuine data scarcity problem, as existing Kannada digit datasets are much smaller. The transparent documentation of the collection process, including volunteer demographics and scanner hardware, is a strength. The main numerical claims are checkable because the code and data are public.
major comments (3)
- [Section 2.1.4, Table 1, Abstract] The headline baseline accuracy is not comparable to MNIST or KMNIST because the main test set is deliberately non-random. Section 2.1.4 states that the 65 volunteers are ranked by a proxy difficulty score (mean probability that a MNIST-trained CNN maps their 0, 3, and 7 to 0, 2, and 2), and the 8 worst volunteers are assigned to the 10,000-image test set. The paper even reports that a random split yields over 99% accuracy, versus 96.85% on the worst-cohort test. MNIST's test set is a random draw from the same population, so the reported 96.85% reflects an adversarial writer cohort, not average generalization to unseen writers. The 'drop-in replacement' framing in the abstract and conclusion is therefore overstated. Please report both random-split and worst-cohort results, report variance across multiple training seeds, and reframe the abstract/conclusion claims accordingly.
- [Section 4 vs. Table 1] There is an internal inconsistency in the headline baseline: Section 4 states 'we achieved 97.13% top-1 accuracy' for the main dataset with a 60,000-10,000 split, while Table 1 reports an accuracy of 0.9685 (96.85%), and the abstract reports 96.8%. If 97.13% is from a different training run or epoch, that must be stated; otherwise the numbers must be reconciled. This is load-bearing because the baseline is one of the paper's main advertised results.
- [Section 2.1.3] The sanity check using a MNIST-trained CNN on classes 0, 3, and 7 is a reasonable heuristic, but it is used to rank volunteers and construct the worst-cohort test split. This makes the proxy difficulty score a central element of the evaluation protocol. The paper should show that this proxy is stable across multiple MNIST-trained models and should discuss what happens if the proxy mis-ranks volunteers. As written, the reported test accuracy depends on the specific pretrained CNN used for the ranking.
minor comments (6)
- [Figure 12 caption] The caption reads 'Morphological comparisons between MNIST and the Fashion-MNIST,' but the text and figure describe comparisons with Kannada-MNIST. The caption should be corrected.
- [Conclusion, paragraph 1] The phrase 'The images in the Dig-MNSIT dataset are noisier' contains a typo: 'MNSIT' should be 'MNIST'.
- [Section 4, first paragraph] The text says 'This pre-trained CNN achieved 76.2% top-1 accuracy on the dig-10k dataset,' but it was not pretrained elsewhere; it is the same CNN trained on the main dataset in this paper. Please clarify the wording to avoid confusion.
- [Section 4, first paragraph] The phrase 'with 60,000-10,000 train-test split' is ambiguous because the split described in Section 2.1.4 is not a random split. Please reference the worst-cohort construction explicitly.
- [Section 2.2 and Section 2.3] Section 2.2 calls the dataset '10k Dig-MNIST' while the actual size is 10,240; the label '10k' is acceptable but should be consistent in all places, especially the abstract.
- [Section 3.1, Figure 12] The analysis reports morphological trait statistics, but it does not state whether the differences are statistically significant or based on a random subset. A brief note on the sample size and whether the morphological differences are robust would improve the comparison.
Circularity Check
No significant circularity: the dataset is an empirical contribution with measured baselines, and the worst-cohort split is a disclosed evaluation choice rather than a fitted prediction.
full rationale
The paper's central contribution is the dissemination of a new handwritten-digit dataset and the reporting of baseline accuracies measured on it. There is no derivation chain in which an output quantity is defined in terms of the quantity it is supposed to predict, nor is any reported accuracy obtained by fitting a parameter and then renaming the fit as a prediction. The MNIST-CNN sanity check (Section 2.1.3) is used as a quality check on the parsing pipeline and as the basis for the disclosed worst-cohort train/test split (Section 2.1.4); the split deliberately selects the eight users ranked hardest by a proxy difficulty score, and the paper explicitly reports that a random split yields over 99% accuracy. This makes the 96.85% test accuracy non-representative of a random writer cohort, but it is still a measured accuracy on a publicly specified test set, not a quantity forced by construction. The self-citations to the authors' prior SAT framework [22] and the 1280-digit dataset appear in related work, a challenge suggestion, and an auxiliary baseline; they are not load-bearing for the central dataset claim. The internal inconsistency between the abstract's 96.8%, Table 1's 96.85%, and Section 4's 97.13% is a correctness or reporting issue, not circularity. Overall, the paper is self-contained as an empirical dataset contribution, and no circular step meets the evidentiary bar of exhibiting a specific reduction or a fitted parameter renamed as a prediction.
Assumptions & free parameters
assumptions (3)
- domain assumption The row index of each cell in the filled 32x40 grid corresponds to the intended digit class label.
- domain assumption The Sudoku-solver-based pre-processing pipeline reliably segments the handwritten digits from the scanned grids.
- ad hoc to paper The MNIST-CNN predictions on classes 0, 3, and 7 are a valid sanity check for the Kannada glyphs.
Cite this review
Pith. "Pith review of Kannada-MNIST: A new handwritten digits dataset for the Kannada language." pith.science (2026). https://pith.science/paper/VJYTAF5L
@misc{pith2026190801242,
author = {Pith},
title = {Pith review of: Kannada-MNIST: A new handwritten digits dataset for the Kannada language},
year = {2026},
howpublished = {\url{https://pith.science/paper/VJYTAF5L}},
note = {Machine review of arXiv:1908.01242}
}
abstract
In this paper, we disseminate a new handwritten digits-dataset, termed Kannada-MNIST, for the Kannada script, that can potentially serve as a direct drop-in replacement for the original MNIST dataset. In addition to this dataset, we disseminate an additional real world handwritten dataset (with $10k$ images), which we term as the Dig-MNIST dataset that can serve as an out-of-domain test dataset. We also duly open source all the code as well as the raw scanned images along with the scanner settings so that researchers who want to try out different signal processing pipelines can perform end-to-end comparisons. We provide high level morphological comparisons with the MNIST dataset and provide baselines accuracies for the dataset disseminated. The initial baselines obtained using an oft-used CNN architecture ($96.8\%$ for the main test-set and $76.1\%$ for the Dig-MNIST test-set) indicate that these datasets do provide a sterner challenge with regards to generalizability than MNIST or the KMNIST datasets. We also hope this dissemination will spur the creation of similar datasets for all the languages that use different symbols for the numeral digits.
Figures
Figures from the paper (20 more)
Forward citations
Cited by 1 Pith paper
-
Privacy-Utility Trade-off in Data Publication: A Bilevel Optimization Framework with Curvature-Guided Perturbation
A bilevel optimization framework with curvature-guided geodesic perturbation reduces membership inference attack success while preserving downstream classification accuracy and sample quality.
Reference graph
Works this paper leans on
-
[1]
Mnist handwritten digit database
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. AT&T Labs [Online]. Available: http://yann. lecun. com/exdb/mnist, 2:18, 2010
work page 2010
-
[2]
Student slang at IIT Madras: a linguistic field study
Evelyn Richter. Student slang at IIT Madras: a linguistic field study. Master’s thesis, Technische Universitat Chemnitz, Str. der Nationen 62, 09111 Chemnitz, Germany, 2006
work page 2006
-
[3]
https://en.wikipedia.org/wiki/Kannada, 2019
Kannada. https://en.wikipedia.org/wiki/Kannada, 2019. [Online; accessed 16-Mar-2019]
work page 2019
-
[4]
https://en.wikipedia.org/wiki/Eighth_Schedule_ to_the_Constitution_of_India, 2019
Eighth schedule to the constitution of india. https://en.wikipedia.org/wiki/Eighth_Schedule_ to_the_Constitution_of_India, 2019. [Online; accessed 16-Mar-2019]
work page 2019
-
[5]
Scholar throws light on the origin, evolution of kannada numerals
Special correspondent;. Scholar throws light on the origin, evolution of kannada numerals. The Hindu, Nov 2017. 9https://github.com/vinayprabhu/Kannada_MNIST/blob/master/colab_notebooks/5) _Synthetic_seed_image_generation.ipynb 9 - AUGUST 6, 2019
work page 2017
-
[6]
Gudnapur inscription of kadamba ravivarma
BR Gopal. Gudnapur inscription of kadamba ravivarma. Srikanthika: Dr S. Srikantha Sastri Felicitation V olume, pages 61–72, 1973
work page 1973
-
[7]
G. S. Gai. In01046 no.22: Plate xxii gudnapur inscription of ravivarman, 1996
work page 1996
-
[8]
M. G. Manjunath and G. K. Devarajaswamy. Kannada lipi vikasa. Technical report, Jagadhguru Sri Madhvacharya Trust, Sri Raghavendra Swami Matta, Mantralaya, 2004
work page 2004
Show all 23 references
-
[9]
https://unicode.org/charts/PDF/U0C80.pdf, 2019
Unicode charts. https://unicode.org/charts/PDF/U0C80.pdf, 2019. [Online; accessed 16-Mar- 2019]
2019
-
[10]
Recognition of handwritten kannada numerals
Nabin Sharma, U Pal, and Fumitaka Kimura. Recognition of handwritten kannada numerals. In 9th International Conference on Information Technology (ICIT’06), pages 133–136. IEEE, 2006
2006
-
[11]
Recognition of isolated handwritten kannada numerals based on image fusion method
GG Rajput and Mallikarjun Hangarge. Recognition of isolated handwritten kannada numerals based on image fusion method. In International Conference on Pattern Recognition and Machine Intelligence , pages 153–160. Springer, 2007
2007
-
[12]
Printed and handwritten mixed kannada numerals recognition using svm
GG Rajput, Rajeswari Horakeri, and Sidramappa Chandrakant. Printed and handwritten mixed kannada numerals recognition using svm. International Journal on Computer Science and Engineering , 2(05):1622–1626, 2010
2010
-
[13]
T. E. de Campos, B. R. Babu, and M. Varma. Character recognition in natural images. In Proceedings of the International Conference on Computer Vision Theory and Applications, Lisbon, Portugal , February 2009
2009
-
[14]
Deep learning approach for recognition of handwritten kannada numerals
Anirudh Ganesh, Ashwin R Jadhav, and KA Cibi Pragadeesh. Deep learning approach for recognition of handwritten kannada numerals. In International Conference on Soft Computing and Pattern Recognition , pages 294–303. Springer, 2016
2016
-
[15]
Cold case: The lost mnist digits
Chhavi Yadav and Léon Bottou. Cold case: The lost mnist digits. Technical report, arxiv-1905.10498, may 2019
1905 arXiv
-
[16]
Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms
Han Xiao, Kashif Rasul, and Roland V ollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. 2017
2017
-
[17]
Deep learning for classical japanese literature
Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. 2018
2018
-
[18]
https://gist.github.com/mineshpatel1/ 209038c64c19d5e78e0a878320797631#file-sudoku_cv-py, 2017
Sudoku solver 2. https://gist.github.com/mineshpatel1/ 209038c64c19d5e78e0a878320797631#file-sudoku_cv-py, 2017. [Online; accessed 16- July-2019]
2017
-
[19]
https://medium.com/@o.kroeger/ tensorflow-mnist-and-your-own-handwritten-digits-4d1cd32bbab4 , 2016
Tensorflow, mnist and your own handwritten digits. https://medium.com/@o.kroeger/ tensorflow-mnist-and-your-own-handwritten-digits-4d1cd32bbab4 , 2016. [Online; ac- cessed 16-July-2019]
2016
-
[20]
Castro, Jeremy Tan, Bernhard Kainz, Ender Konukoglu, and Ben Glocker
Daniel C. Castro, Jeremy Tan, Bernhard Kainz, Ender Konukoglu, and Ben Glocker. Morpho-MNIST: Quantitative assessment and diagnostics for representation learning. 2018
2018
-
[21]
Umap: Uniform manifold approximation and projection for dimension reduction
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426, 2018
2018 arXiv
-
[22]
Fonts-2-handwriting: A seed-augment-train framework for universal digit classification
Vinay Uday Prabhu, Sanghyun Han, Dian Ang Yap, Mihail Douhaniaris, Preethi Seshadri, and John Wha- ley. Fonts-2-handwriting: A seed-augment-train framework for universal digit classification. arXiv preprint arXiv:1905.08633, 2019
1905 arXiv
-
[23]
Biomedical image augmentation using augmentor
Marcus D Bloice, Peter M Roth, and Andreas Holzinger. Biomedical image augmentation using augmentor. Bioinformatics, 2019. 10 - AUGUST 6, 2019 Figure 7: The main dataset creation workflow 11 - AUGUST 6, 2019 Figure 8: Class-wise mean images of the train set, the test set and th...
2019
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.