Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Kannada-MNIST: A new handwritten digits dataset for the Kannada language

T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read Kannada-MNIST is a 60,000-image handwritten digit dataset in the Kannada script, designed as a drop-in MNIST replacement, whose worst-writer test set drops a standard CNN to 96.85% and an out-of-domain set to 76.2%.

desk verdict A genuinely useful Kannada digit dataset with open-sourced raw scans and code, but the 'drop-in replacement' framing overreaches because the headline accuracy is measured on a deliberately adversarial test cohort, not a random sample. read the letter →

arxiv 1908.01242 v1 pith:VJYTAF5L submitted 2019-08-03 cs.CV cs.LGstat.ML

classification cs.CVcs.LGstat.ML
keywords handwrittendigitrecognitionKannadascriptMNISTdatasetbenchmarkout-of-domaingeneralizationwriter-independentevaluationdomainadaptationconvolutionalneuralnetwork
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper creates Kannada-MNIST, a 60,000-train and 10,000-test handwritten digit dataset for the Kannada script, formatted as a direct MNIST replacement. It also releases Dig-MNIST, a 10,240-image set written by people encountering the script for the first time, to serve as an out-of-domain test. The central result is that a standard CNN reaches 96.85% accuracy on the main dataset but only 76.2% on Dig-MNIST, and the paper argues this gap exposes genuine cross-writer generalization difficulty rather than simple glyph recognition. A sympathetic reader would care because this gives the machine learning community a reproducible, openly documented benchmark for testing how classifiers behave when writers are unseen.

What carries the argument

The load-bearing object is the worst-cohort split rule. Each of the 65 volunteers is scored by the mean probability that a MNIST-trained CNN classifies their Kannada 0 as 0 and their Kannada 3 and 7 as 2; the eight highest-scoring volunteers form the 10,000-image test set and the next 47 form the 60,000-image training set. This transforms the dataset from a within-writer recognition benchmark into a cross-writer generalization benchmark. The second object is Dig-MNIST, a 10,240-image out-of-domain test set written under different conditions by people new to the Kannada script, which provides a harder generalization target.

What would settle it

Train a CNN on 57 of the 65 volunteers and test on a randomly chosen eighth volunteer's digits instead of the worst-scoring cohort; if accuracy lands near the above-99% figure the paper saw in random splits, the 96.85% headline is an artifact of the selection rule. Inspecting per-writer accuracy inside the worst cohort would also show whether the difficulty is spread across all eight writers or driven by one or two.

Watch

Extended reading notes

Core claim

The paper's central claim is that a large, MNIST-compatible handwritten Kannada digit dataset can be built in the wild, and that the dominating challenge it poses is generalization to unseen writers, not recognition of familiar glyphs. Random splits between writers already reached above 99% accuracy, so the authors deliberately constructed the test set from the eight volunteers whose Kannada digits a MNIST-trained CNN found hardest to map to similar-looking MNIST classes; on that worst-cohort test set accuracy fell to 96.85%. On Dig-MNIST, collected with different writers, different paper, different scanner settings, and many first-time users of the script, the same model fell to 76.17%. Because the dataset, raw scans, and parsing code are all released, these numbers can be reproduced and the failure modes inspected.

Load-bearing premise

The load-bearing premise is that the worst-cohort selection rule—ranking volunteers by how often a MNIST-trained CNN maps their Kannada 0, 3, and 7 to the similar-looking MNIST classes 0 and 2—ranks true out-of-sample generalization difficulty, so the 96.85% figure describes what a classifier should expect on a new writer.

Editorial extensions

If this is right

  • The reported 96.85% and 76.2% accuracies become reference baselines that any future Kannada digit classifier must beat.
  • Dig-MNIST turns out-of-domain generalization into a measurable task: improve on 76.2% after training only on Kannada-MNIST, without image preprocessing, which the paper poses as an explicit open challenge.
  • Because raw scans, scanner settings, and parsing code are released, researchers can test whether different segmentation pipelines change accuracy independently of the classifier.
  • The quantitative comparisons—less peaked mean intensity, lower variance explained by the top PCA components, and different morphological trait distributions—give users a concrete sense of how Kannada-MNIST differs from MNIST.
  • The released synthetic font seed code allows others to generate large synthetic Kannada digit sets, which the paper suggests as a route toward higher Dig-MNIST accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because random writer splits already give above 99% accuracy, the 96.85% headline should be read as a property of the worst-cohort selection rule, not of Kannada handwriting per se; the Dig-MNIST result is the cleaner measure of script-specific difficulty.
  • Editorial inference: the Dig-MNIST drop may be partly caused by collection artifacts the paper documents—volunteers overrunning grid lines, smudges, and grid borders—so an ablation that removes partial strokes and border noise would separate segmentation artifacts from true out-of-domain glyph variation.
  • Editorial inference: applying the same worst-cohort proxy to the original MNIST writers would provide a missing control, showing whether writer-split difficulty drops even for familiar Arabic numerals or is sharpened by Kannada's confusable 0, 3, and 7 glyphs.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces Kannada-MNIST, a handwritten Kannada digits dataset of 60,000 training and 10,000 test images in MNIST-compatible 28x28 grayscale format, along with a separate 10,240-image Dig-MNIST out-of-domain test set. The authors document the data collection pipeline, including volunteer recruitment, scanning, grid segmentation, MNIST-ization, and a sanity check using a CNN pretrained on MNIST digits for the visually similar glyphs 0, 3, and 7. They provide morphological and dimensionality-reduction comparisons with MNIST and report CNN baselines: 96.85% accuracy on the main test set and 76.17% on Dig-MNIST. The central claim is that Kannada-MNIST can serve as a direct drop-in replacement for the original MNIST dataset and provides a sterner generalization challenge.

Significance. If the evaluation protocol is clarified, this is a valuable community resource: raw scans, scanner settings, preprocessing code, and model code are all open-sourced, which makes the dataset creation pipeline fully reproducible and supports end-to-end research on handwritten digit recognition in an Indic script. The paper also addresses a genuine data scarcity problem, as existing Kannada digit datasets are much smaller. The transparent documentation of the collection process, including volunteer demographics and scanner hardware, is a strength. The main numerical claims are checkable because the code and data are public.

major comments (3)
  1. [Section 2.1.4, Table 1, Abstract] The headline baseline accuracy is not comparable to MNIST or KMNIST because the main test set is deliberately non-random. Section 2.1.4 states that the 65 volunteers are ranked by a proxy difficulty score (mean probability that a MNIST-trained CNN maps their 0, 3, and 7 to 0, 2, and 2), and the 8 worst volunteers are assigned to the 10,000-image test set. The paper even reports that a random split yields over 99% accuracy, versus 96.85% on the worst-cohort test. MNIST's test set is a random draw from the same population, so the reported 96.85% reflects an adversarial writer cohort, not average generalization to unseen writers. The 'drop-in replacement' framing in the abstract and conclusion is therefore overstated. Please report both random-split and worst-cohort results, report variance across multiple training seeds, and reframe the abstract/conclusion claims accordingly.
  2. [Section 4 vs. Table 1] There is an internal inconsistency in the headline baseline: Section 4 states 'we achieved 97.13% top-1 accuracy' for the main dataset with a 60,000-10,000 split, while Table 1 reports an accuracy of 0.9685 (96.85%), and the abstract reports 96.8%. If 97.13% is from a different training run or epoch, that must be stated; otherwise the numbers must be reconciled. This is load-bearing because the baseline is one of the paper's main advertised results.
  3. [Section 2.1.3] The sanity check using a MNIST-trained CNN on classes 0, 3, and 7 is a reasonable heuristic, but it is used to rank volunteers and construct the worst-cohort test split. This makes the proxy difficulty score a central element of the evaluation protocol. The paper should show that this proxy is stable across multiple MNIST-trained models and should discuss what happens if the proxy mis-ranks volunteers. As written, the reported test accuracy depends on the specific pretrained CNN used for the ranking.
minor comments (6)
  1. [Figure 12 caption] The caption reads 'Morphological comparisons between MNIST and the Fashion-MNIST,' but the text and figure describe comparisons with Kannada-MNIST. The caption should be corrected.
  2. [Conclusion, paragraph 1] The phrase 'The images in the Dig-MNSIT dataset are noisier' contains a typo: 'MNSIT' should be 'MNIST'.
  3. [Section 4, first paragraph] The text says 'This pre-trained CNN achieved 76.2% top-1 accuracy on the dig-10k dataset,' but it was not pretrained elsewhere; it is the same CNN trained on the main dataset in this paper. Please clarify the wording to avoid confusion.
  4. [Section 4, first paragraph] The phrase 'with 60,000-10,000 train-test split' is ambiguous because the split described in Section 2.1.4 is not a random split. Please reference the worst-cohort construction explicitly.
  5. [Section 2.2 and Section 2.3] Section 2.2 calls the dataset '10k Dig-MNIST' while the actual size is 10,240; the label '10k' is acceptable but should be consistent in all places, especially the abstract.
  6. [Section 3.1, Figure 12] The analysis reports morphological trait statistics, but it does not state whether the differences are statistically significant or based on a random subset. A brief note on the sample size and whether the morphological differences are robust would improve the comparison.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the dataset is an empirical contribution with measured baselines, and the worst-cohort split is a disclosed evaluation choice rather than a fitted prediction.

full rationale

The paper's central contribution is the dissemination of a new handwritten-digit dataset and the reporting of baseline accuracies measured on it. There is no derivation chain in which an output quantity is defined in terms of the quantity it is supposed to predict, nor is any reported accuracy obtained by fitting a parameter and then renaming the fit as a prediction. The MNIST-CNN sanity check (Section 2.1.3) is used as a quality check on the parsing pipeline and as the basis for the disclosed worst-cohort train/test split (Section 2.1.4); the split deliberately selects the eight users ranked hardest by a proxy difficulty score, and the paper explicitly reports that a random split yields over 99% accuracy. This makes the 96.85% test accuracy non-representative of a random writer cohort, but it is still a measured accuracy on a publicly specified test set, not a quantity forced by construction. The self-citations to the authors' prior SAT framework [22] and the 1280-digit dataset appear in related work, a challenge suggestion, and an auxiliary baseline; they are not load-bearing for the central dataset claim. The internal inconsistency between the abstract's 96.8%, Table 1's 96.85%, and Section 4's 97.13% is a correctness or reporting issue, not circularity. Overall, the paper is self-contained as an empirical dataset contribution, and no circular step meets the evidentiary bar of exhibiting a specific reduction or a fitted parameter renamed as a prediction.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The dataset construction relies on two domain assumptions: (1) grid row indices provide correct labels, and (2) the Sudoku-based segmentation reliably extracts digits. No free parameters are fit to data; CNN hyperparameters come from the standard Keras MNIST example.

assumptions (3)
  • domain assumption The row index of each cell in the filled 32x40 grid corresponds to the intended digit class label.
    Labels are assigned from grid row index without per-image human verification; see Section 2.1.2.
  • domain assumption The Sudoku-solver-based pre-processing pipeline reliably segments the handwritten digits from the scanned grids.
    Section 2.1.2 relies on the pipeline from [18] to crop, warp and slice the grid images.
  • ad hoc to paper The MNIST-CNN predictions on classes 0, 3, and 7 are a valid sanity check for the Kannada glyphs.
    Section 2.1.3 uses a CNN pre-trained on MNIST to check glyph similarity; this assumes transfer of visual similarity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Kannada-MNIST: A new handwritten digits dataset for the Kannada language." pith.science (2026). https://pith.science/paper/VJYTAF5L

@misc{pith2026190801242,
  author       = {Pith},
  title        = {Pith review of: Kannada-MNIST: A new handwritten digits dataset for the Kannada language},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VJYTAF5L}},
  note         = {Machine review of arXiv:1908.01242}
}
abstract

In this paper, we disseminate a new handwritten digits-dataset, termed Kannada-MNIST, for the Kannada script, that can potentially serve as a direct drop-in replacement for the original MNIST dataset. In addition to this dataset, we disseminate an additional real world handwritten dataset (with $10k$ images), which we term as the Dig-MNIST dataset that can serve as an out-of-domain test dataset. We also duly open source all the code as well as the raw scanned images along with the scanner settings so that researchers who want to try out different signal processing pipelines can perform end-to-end comparisons. We provide high level morphological comparisons with the MNIST dataset and provide baselines accuracies for the dataset disseminated. The initial baselines obtained using an oft-used CNN architecture ($96.8\%$ for the main test-set and $76.1\%$ for the Dig-MNIST test-set) indicate that these datasets do provide a sterner challenge with regards to generalizability than MNIST or the KMNIST datasets. We also hope this dissemination will spur the creation of similar datasets for all the languages that use different symbols for the numeral digits.

Figures

Figures reproduced from arXiv: 1908.01242 by the authors.

Figure 1
Figure 1. Usage of the Kannada numerals on vehicular license plates [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Evolution of the Kannada numerals through the ages ([8]) [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The character code tables for Kannada-MNIST from the Unicode Standard, Version 12.1 [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (20 more)
Figure 4
Figure 4. Figure 4: MNIST-ized renderings of the 0-9 Kannada numerals in 11 modern fonts [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: The similarity between the glyphs for 3 and 7 in Kannada [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: Deformations in the numeral glyphs for 6 [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: The main dataset creation workflow 11 [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Class-wise mean images of the train set, the test set and the difference between the means of the train and test [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Preparing the dig-dataset in Redwood City [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: User-wise class-wise mean images for the Dig-10k dataset [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]
Figure 11
Figure 11. Figure 11: Mean pixel-wise intensities comparisons between MNIST and the Kannada-MNIST datasets [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]
Figure 12
Figure 12. Figure 12: Morphological comparisons between MNIST and the Fashion-MNIST [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]
Figure 13
Figure 13. Figure 13: PCA analysis for the two datasets [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]
Figure 14
Figure 14. Figure 14: Two-dimensional Uniform Manifold Approximation and Projection (UMAP) plots for the two datasets [PITH_FULL_IMAGE:figures/full_fig_p015_14.png]
Figure 15
Figure 15. Figure 15: The CNN architecture used in the paper [PITH_FULL_IMAGE:figures/full_fig_p016_15.png]
Figure 16
Figure 16. Figure 16: Confusion matrix with regards to the Kannada-MNIST datasets [PITH_FULL_IMAGE:figures/full_fig_p016_16.png]
Figure 17
Figure 17. Figure 17: Train and test accuracies for the CNN trained and tested on the [PITH_FULL_IMAGE:figures/full_fig_p017_17.png]
Figure 18
Figure 18. Figure 18: Confusion matrix for the dig-10k dataset 17 [PITH_FULL_IMAGE:figures/full_fig_p017_18.png]
Figure 19
Figure 19. Figure 19: Un-normalized confusion matrix for the 1280 digits dataset using in [22] [PITH_FULL_IMAGE:figures/full_fig_p018_19.png]
Figure 20
Figure 20. Figure 20: Images belonging to class-0 in the 1280-digits dataset that were misclassified by the CNN trained on the [PITH_FULL_IMAGE:figures/full_fig_p019_20.png]
Figure 21
Figure 21. Figure 21: Images belonging to class-7 in the 1280-digits dataset that were misclassified by the CNN trained on the [PITH_FULL_IMAGE:figures/full_fig_p020_21.png]
Figure 22
Figure 22. Figure 22: Images belonging to class-8 in the 1280-digits dataset that were misclassified by the CNN trained on the [PITH_FULL_IMAGE:figures/full_fig_p020_22.png]
Figure 23
Figure 23. Figure 23: Photos of hard copies of the handwritten sheets for the two datasets [PITH_FULL_IMAGE:figures/full_fig_p021_23.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Privacy-Utility Trade-off in Data Publication: A Bilevel Optimization Framework with Curvature-Guided Perturbation

    cs.LG 2025-09 conditional novelty 6.0 of 10

    A bilevel optimization framework with curvature-guided geodesic perturbation reduces membership inference attack success while preserving downstream classification accuracy and sample quality.

Reference graph

Works this paper leans on

23 extracted references · 21 canonical work pages · cited by 1 Pith paper

  1. [1]

    Mnist handwritten digit database

    Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. AT&T Labs [Online]. Available: http://yann. lecun. com/exdb/mnist, 2:18, 2010

  2. [2]

    Student slang at IIT Madras: a linguistic field study

    Evelyn Richter. Student slang at IIT Madras: a linguistic field study. Master’s thesis, Technische Universitat Chemnitz, Str. der Nationen 62, 09111 Chemnitz, Germany, 2006

  3. [3]

    https://en.wikipedia.org/wiki/Kannada, 2019

    Kannada. https://en.wikipedia.org/wiki/Kannada, 2019. [Online; accessed 16-Mar-2019]

  4. [4]

    https://en.wikipedia.org/wiki/Eighth_Schedule_ to_the_Constitution_of_India, 2019

    Eighth schedule to the constitution of india. https://en.wikipedia.org/wiki/Eighth_Schedule_ to_the_Constitution_of_India, 2019. [Online; accessed 16-Mar-2019]

  5. [5]

    Scholar throws light on the origin, evolution of kannada numerals

    Special correspondent;. Scholar throws light on the origin, evolution of kannada numerals. The Hindu, Nov 2017. 9https://github.com/vinayprabhu/Kannada_MNIST/blob/master/colab_notebooks/5) _Synthetic_seed_image_generation.ipynb 9 - AUGUST 6, 2019

  6. [6]

    Gudnapur inscription of kadamba ravivarma

    BR Gopal. Gudnapur inscription of kadamba ravivarma. Srikanthika: Dr S. Srikantha Sastri Felicitation V olume, pages 61–72, 1973

  7. [7]

    G. S. Gai. In01046 no.22: Plate xxii gudnapur inscription of ravivarman, 1996

  8. [8]

    M. G. Manjunath and G. K. Devarajaswamy. Kannada lipi vikasa. Technical report, Jagadhguru Sri Madhvacharya Trust, Sri Raghavendra Swami Matta, Mantralaya, 2004

Show all 23 references
  1. [9]

    https://unicode.org/charts/PDF/U0C80.pdf, 2019

    Unicode charts. https://unicode.org/charts/PDF/U0C80.pdf, 2019. [Online; accessed 16-Mar- 2019]

  2. [10]

    Recognition of handwritten kannada numerals

    Nabin Sharma, U Pal, and Fumitaka Kimura. Recognition of handwritten kannada numerals. In 9th International Conference on Information Technology (ICIT’06), pages 133–136. IEEE, 2006

  3. [11]

    Recognition of isolated handwritten kannada numerals based on image fusion method

    GG Rajput and Mallikarjun Hangarge. Recognition of isolated handwritten kannada numerals based on image fusion method. In International Conference on Pattern Recognition and Machine Intelligence , pages 153–160. Springer, 2007

  4. [12]

    Printed and handwritten mixed kannada numerals recognition using svm

    GG Rajput, Rajeswari Horakeri, and Sidramappa Chandrakant. Printed and handwritten mixed kannada numerals recognition using svm. International Journal on Computer Science and Engineering , 2(05):1622–1626, 2010

  5. [13]

    T. E. de Campos, B. R. Babu, and M. Varma. Character recognition in natural images. In Proceedings of the International Conference on Computer Vision Theory and Applications, Lisbon, Portugal , February 2009

  6. [14]

    Deep learning approach for recognition of handwritten kannada numerals

    Anirudh Ganesh, Ashwin R Jadhav, and KA Cibi Pragadeesh. Deep learning approach for recognition of handwritten kannada numerals. In International Conference on Soft Computing and Pattern Recognition , pages 294–303. Springer, 2016

  7. [15]

    Cold case: The lost mnist digits

    Chhavi Yadav and Léon Bottou. Cold case: The lost mnist digits. Technical report, arxiv-1905.10498, may 2019

  8. [16]

    Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms

    Han Xiao, Kashif Rasul, and Roland V ollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. 2017

  9. [17]

    Deep learning for classical japanese literature

    Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. 2018

  10. [18]

    https://gist.github.com/mineshpatel1/ 209038c64c19d5e78e0a878320797631#file-sudoku_cv-py, 2017

    Sudoku solver 2. https://gist.github.com/mineshpatel1/ 209038c64c19d5e78e0a878320797631#file-sudoku_cv-py, 2017. [Online; accessed 16- July-2019]

  11. [19]

    https://medium.com/@o.kroeger/ tensorflow-mnist-and-your-own-handwritten-digits-4d1cd32bbab4 , 2016

    Tensorflow, mnist and your own handwritten digits. https://medium.com/@o.kroeger/ tensorflow-mnist-and-your-own-handwritten-digits-4d1cd32bbab4 , 2016. [Online; ac- cessed 16-July-2019]

  12. [20]

    Castro, Jeremy Tan, Bernhard Kainz, Ender Konukoglu, and Ben Glocker

    Daniel C. Castro, Jeremy Tan, Bernhard Kainz, Ender Konukoglu, and Ben Glocker. Morpho-MNIST: Quantitative assessment and diagnostics for representation learning. 2018

  13. [21]

    Umap: Uniform manifold approximation and projection for dimension reduction

    Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426, 2018

  14. [22]

    Fonts-2-handwriting: A seed-augment-train framework for universal digit classification

    Vinay Uday Prabhu, Sanghyun Han, Dian Ang Yap, Mihail Douhaniaris, Preethi Seshadri, and John Wha- ley. Fonts-2-handwriting: A seed-augment-train framework for universal digit classification. arXiv preprint arXiv:1905.08633, 2019

  15. [23]

    Biomedical image augmentation using augmentor

    Marcus D Bloice, Peter M Roth, and Andreas Holzinger. Biomedical image augmentation using augmentor. Bioinformatics, 2019. 10 - AUGUST 6, 2019 Figure 7: The main dataset creation workflow 11 - AUGUST 6, 2019 Figure 8: Class-wise mean images of the train set, the test set and th...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.