{"id":"ce43ef3d-d51a-4089-8456-e2d645af7b13","arxiv_id":"1908.01242","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new, fully open-source handwritten Kannada digits dataset with MNIST-compatible format and baseline CNN accuracies of about 97% in-domain and 76% out-of-domain.","lead":"This paper introduces Kannada-MNIST, a new handwritten-digit dataset for the Kannada script with 60,000 training and 10,000 test images, plus a harder out-of-domain set called Dig-MNIST. It is a format-compatible alternative to MNIST for benchmarking handwriting recognition and domain generalization.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 96.85% accuracy depends on a deliberately non-random worst-cohort test split (Sec. 2.1.4); without a random-split comparison or cross-seed variance, the drop-in replacement claim overstates comparability to MNIST.","rationale":"The reader's weakest assumption points to the worst-cohort test selection, and my reading agrees that this is the most load-bearing issue for the paper's central claim. The dataset itself is real, the raw scans are released, and the pipeline is documented, but the numerical headline and the drop-in replacement framing are calibrated against a deliberately hard, proxy-selected test set. That design choice is disclosed, so it is not a soundness failure; it is a comparability and representativeness problem. The paper should either justify why the worst-cohort split is the intended benchmark or provide random-split results and seed variance so users can interpret the 96.85 percent figure correctly. The secondary inconsistency between the Section 4 text and Table 1 is concrete but minor; it reinforces the need for care. Because the reader already issued a CONDITIONAL verdict and my concern supports the same condition, I recommend keeping that verdict unchanged.","tokens_in":9078,"tokens_out":4192,"duration_ms":45042,"concrete_test":"Reconstruct a random-split evaluation from the released raw scans: randomly assign volunteers to a 60k/10k train/test split, train the same Keras MNIST CNN with at least five different seeds, and report mean plus-or-minus standard deviation of test accuracy. If random-split accuracy is close to the paper's stated 99 percent, the 96.85 percent headline is an artifact of worst-cohort selection and the drop-in replacement comparison should be reframed. If random-split accuracy is also near 97 percent, the concern loses force. Independently, re-read Table 1 against the Section 4 text to resolve whether the headline baseline is 96.85 or 97.13 percent.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Kannada-MNIST is a drop-in replacement for MNIST rests on the default test set being comparable to MNIST's random test set. It is not. Section 2.1.4 ranks the 65 volunteers by a proxy difficulty score (mean probability that a MNIST-trained CNN maps their 0, 3, and 7 to 0, 2, and 2, respectively) and assigns the 8 hardest users to the 10k test set, with the remaining 47 users forming the 60k train set. The paper even reports that a random split yields over 99 percent accuracy (Section 2.1.4), versus 96.85 percent on the worst-cohort test (Table 1). Consequently the reported 96.85 percent reflects a deliberately adversarial subset of writers, not the average generalization performance on a random writer cohort. This makes the headline comparison to MNIST and KMNIST misleading: MNIST's test set is a random draw from the same population, whereas here the train and test sets are separated subpopulations, with the split chosen using a proxy based only on digits 0, 3, and 7. The concern is not that the split is hidden, but that the drop-in framing and the numeric baselines inherit a non-standard evaluation protocol. There is also a smaller internal inconsistency: Section 4 states 97.13 percent top-1 accuracy while Table 1 reports 96.85 percent, and the abstract says 96.8 percent; the headline baseline is not internally consistent.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Kannada-MNIST, a handwritten Kannada digits dataset of 60,000 training and 10,000 test images in MNIST-compatible 28x28 grayscale format, along with a separate 10,240-image Dig-MNIST out-of-domain test set. The authors document the data collection pipeline, including volunteer recruitment, scanning, grid segmentation, MNIST-ization, and a sanity check using a CNN pretrained on MNIST digits for the visually similar glyphs 0, 3, and 7. They provide morphological and dimensionality-reduction comparisons with MNIST and report CNN baselines: 96.85% accuracy on the main test set and 76.17% on Dig-MNIST. The central claim is that Kannada-MNIST can serve as a direct drop-in replacement for the original MNIST dataset and provides a sterner generalization challenge.","tokens_in":9456,"tokens_out":3306,"duration_ms":29805,"significance":"If the evaluation protocol is clarified, this is a valuable community resource: raw scans, scanner settings, preprocessing code, and model code are all open-sourced, which makes the dataset creation pipeline fully reproducible and supports end-to-end research on handwritten digit recognition in an Indic script. The paper also addresses a genuine data scarcity problem, as existing Kannada digit datasets are much smaller. The transparent documentation of the collection process, including volunteer demographics and scanner hardware, is a strength. The main numerical claims are checkable because the code and data are public.","major_comments":[{"comment":"The headline baseline accuracy is not comparable to MNIST or KMNIST because the main test set is deliberately non-random. Section 2.1.4 states that the 65 volunteers are ranked by a proxy difficulty score (mean probability that a MNIST-trained CNN maps their 0, 3, and 7 to 0, 2, and 2), and the 8 worst volunteers are assigned to the 10,000-image test set. The paper even reports that a random split yields over 99% accuracy, versus 96.85% on the worst-cohort test. MNIST's test set is a random draw from the same population, so the reported 96.85% reflects an adversarial writer cohort, not average generalization to unseen writers. The 'drop-in replacement' framing in the abstract and conclusion is therefore overstated. Please report both random-split and worst-cohort results, report variance across multiple training seeds, and reframe the abstract/conclusion claims accordingly.","section":"Section 2.1.4, Table 1, Abstract"},{"comment":"There is an internal inconsistency in the headline baseline: Section 4 states 'we achieved 97.13% top-1 accuracy' for the main dataset with a 60,000-10,000 split, while Table 1 reports an accuracy of 0.9685 (96.85%), and the abstract reports 96.8%. If 97.13% is from a different training run or epoch, that must be stated; otherwise the numbers must be reconciled. This is load-bearing because the baseline is one of the paper's main advertised results.","section":"Section 4 vs. Table 1"},{"comment":"The sanity check using a MNIST-trained CNN on classes 0, 3, and 7 is a reasonable heuristic, but it is used to rank volunteers and construct the worst-cohort test split. This makes the proxy difficulty score a central element of the evaluation protocol. The paper should show that this proxy is stable across multiple MNIST-trained models and should discuss what happens if the proxy mis-ranks volunteers. As written, the reported test accuracy depends on the specific pretrained CNN used for the ranking.","section":"Section 2.1.3"}],"minor_comments":[{"comment":"The caption reads 'Morphological comparisons between MNIST and the Fashion-MNIST,' but the text and figure describe comparisons with Kannada-MNIST. The caption should be corrected.","section":"Figure 12 caption"},{"comment":"The phrase 'The images in the Dig-MNSIT dataset are noisier' contains a typo: 'MNSIT' should be 'MNIST'.","section":"Conclusion, paragraph 1"},{"comment":"The text says 'This pre-trained CNN achieved 76.2% top-1 accuracy on the dig-10k dataset,' but it was not pretrained elsewhere; it is the same CNN trained on the main dataset in this paper. Please clarify the wording to avoid confusion.","section":"Section 4, first paragraph"},{"comment":"The phrase 'with 60,000-10,000 train-test split' is ambiguous because the split described in Section 2.1.4 is not a random split. Please reference the worst-cohort construction explicitly.","section":"Section 4, first paragraph"},{"comment":"Section 2.2 calls the dataset '10k Dig-MNIST' while the actual size is 10,240; the label '10k' is acceptable but should be consistent in all places, especially the abstract.","section":"Section 2.2 and Section 2.3"},{"comment":"The analysis reports morphological trait statistics, but it does not state whether the differences are statistically significant or based on a random subset. A brief note on the sample size and whether the morphological differences are robust would improve the comparison.","section":"Section 3.1, Figure 12"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take on arXiv:1908.01242 (Kannada-MNIST). The dataset itself is a real contribution: prior Kannada digit resources topped out around 25 samples per class (Chars74k), and this gives 60k/10k plus a 10k out-of-domain Dig-MNIST set, with open-sourced raw scans, segmentation code, and a documented pipeline. The morphological comparisons to MNIST are a nice touch. That earns real credit.\n\nThe main issue is the framing. The test set is not a random sample: Section 2.1.4 ranks the 65 volunteers by a proxy difficulty score computed from a MNIST-trained CNN and puts the worst 8 into the test cohort. The paper is transparent about this, and even reports that a random split gives >99% accuracy. So the reported 96.85% is a deliberate worst-case generalization number, not an MNIST-comparable number. Calling the dataset a 'drop-in replacement' for MNIST sets up an apples-to-oranges comparison: MNIST's test set is a random draw from the same population, while this one is a writer-disjoint adversarial cohort. The right fix is to report both numbers prominently and position the worst-cohort split as a feature, a harder generalization benchmark, rather than a drop-in equivalent.\n\nThere is also a small internal inconsistency: Section 4 says 97.13% top-1 accuracy, Table 1 says 96.85%, and the abstract says 96.8%. Somebody needs to make the numbers agree. Add a cross-seed variance estimate for the baseline and that part is solid.\n\nIf I were refereeing, I'd ask for the random-split baseline to be reported in the main table, a consistency fix, and more careful wording about drop-in equivalence. None of these are fatal; they are packaging issues around a dataset that is genuinely useful.\n\nThis paper deserves a serious referee. The dataset is checkable, the raw scans are there, and the community working on Indic OCR and domain adaptation will get immediate value. I'd cite it.","headline":"A genuinely useful Kannada digit dataset with open-sourced raw scans and code, but the 'drop-in replacement' framing overreaches because the headline accuracy is measured on a deliberately adversarial test cohort, not a random sample.","tokens_in":9887,"tokens_out":2247,"would_cite":true,"duration_ms":21710,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Kannada-MNIST is a 60,000-image handwritten digit dataset in the Kannada script, designed as a drop-in MNIST replacement, whose worst-writer test set drops a standard CNN to 96.85% and an out-of-domain set to 76.2%.","keywords":["handwritten digit recognition","Kannada script","MNIST","dataset benchmark","out-of-domain generalization","writer-independent evaluation","domain adaptation","convolutional neural network"],"falsifier":"Train a CNN on 57 of the 65 volunteers and test on a randomly chosen eighth volunteer's digits instead of the worst-scoring cohort; if accuracy lands near the above-99% figure the paper saw in random splits, the 96.85% headline is an artifact of the selection rule. Inspecting per-writer accuracy inside the worst cohort would also show whether the difficulty is spread across all eight writers or driven by one or two.","tokens_in":8923,"feed_emoji":"✍️","tokens_out":9002,"duration_ms":91268,"temperature":0.7,"pith_summary":"This paper creates Kannada-MNIST, a 60,000-train and 10,000-test handwritten digit dataset for the Kannada script, formatted as a direct MNIST replacement. It also releases Dig-MNIST, a 10,240-image set written by people encountering the script for the first time, to serve as an out-of-domain test. The central result is that a standard CNN reaches 96.85% accuracy on the main dataset but only 76.2% on Dig-MNIST, and the paper argues this gap exposes genuine cross-writer generalization difficulty rather than simple glyph recognition. A sympathetic reader would care because this gives the machine learning community a reproducible, openly documented benchmark for testing how classifiers behave when writers are unseen.","feed_headline":"New Kannada digit set gives MNIST a harder writer-split test","feed_subtitle":"A standard CNN hits 96.85% in-domain but only 76.2% on unfamiliar handwriting.","key_machinery":"The load-bearing object is the worst-cohort split rule. Each of the 65 volunteers is scored by the mean probability that a MNIST-trained CNN classifies their Kannada 0 as 0 and their Kannada 3 and 7 as 2; the eight highest-scoring volunteers form the 10,000-image test set and the next 47 form the 60,000-image training set. This transforms the dataset from a within-writer recognition benchmark into a cross-writer generalization benchmark. The second object is Dig-MNIST, a 10,240-image out-of-domain test set written under different conditions by people new to the Kannada script, which provides a harder generalization target.","core_discovery":"The paper's central claim is that a large, MNIST-compatible handwritten Kannada digit dataset can be built in the wild, and that the dominating challenge it poses is generalization to unseen writers, not recognition of familiar glyphs. Random splits between writers already reached above 99% accuracy, so the authors deliberately constructed the test set from the eight volunteers whose Kannada digits a MNIST-trained CNN found hardest to map to similar-looking MNIST classes; on that worst-cohort test set accuracy fell to 96.85%. On Dig-MNIST, collected with different writers, different paper, different scanner settings, and many first-time users of the script, the same model fell to 76.17%. Because the dataset, raw scans, and parsing code are all released, these numbers can be reproduced and the failure modes inspected.","pith_inferences":["Editorial inference: because random writer splits already give above 99% accuracy, the 96.85% headline should be read as a property of the worst-cohort selection rule, not of Kannada handwriting per se; the Dig-MNIST result is the cleaner measure of script-specific difficulty.","Editorial inference: the Dig-MNIST drop may be partly caused by collection artifacts the paper documents—volunteers overrunning grid lines, smudges, and grid borders—so an ablation that removes partial strokes and border noise would separate segmentation artifacts from true out-of-domain glyph variation.","Editorial inference: applying the same worst-cohort proxy to the original MNIST writers would provide a missing control, showing whether writer-split difficulty drops even for familiar Arabic numerals or is sharpened by Kannada's confusable 0, 3, and 7 glyphs."],"forward_implications":["The reported 96.85% and 76.2% accuracies become reference baselines that any future Kannada digit classifier must beat.","Dig-MNIST turns out-of-domain generalization into a measurable task: improve on 76.2% after training only on Kannada-MNIST, without image preprocessing, which the paper poses as an explicit open challenge.","Because raw scans, scanner settings, and parsing code are released, researchers can test whether different segmentation pipelines change accuracy independently of the classifier.","The quantitative comparisons—less peaked mean intensity, lower variance explained by the top PCA components, and different morphological trait distributions—give users a concrete sense of how Kannada-MNIST differs from MNIST.","The released synthetic font seed code allows others to generate large synthetic Kannada digit sets, which the paper suggests as a route toward higher Dig-MNIST accuracy."],"supporting_citations":[{"why":"It is the standard MNIST handwritten digit benchmark that Kannada-MNIST is designed to replace, supplying the 28x28 format and the comparison baseline.","marker":"[1]"},{"why":"It is the current largest Kannada character dataset used in academic literature, with only 25 samples per numeral, establishing the paucity this paper addresses.","marker":"[13]"},{"why":"It is a prior drop-in replacement for MNIST, establishing the format and use-case that Kannada-MNIST follows.","marker":"[16]"},{"why":"It is a prior MNIST-format digit benchmark in a non-Latin script, used as a comparator for the generalization challenge.","marker":"[17]"},{"why":"It supplies the grid-image preprocessing and slicing pipeline used to extract individual digits from scanned sheets.","marker":"[18]"},{"why":"It supplies the MNIST-ization procedure: thresholding, padding, and center-of-mass shifting used to produce 28x28 images.","marker":"[19]"},{"why":"It is the prior synthetic-to-real Kannada digit framework that supplies the font-based seed generation approach and the 1280-digit real-world test set used as a third evaluation set.","marker":"[22]"}],"fun_headline_variants":["Kannada-MNIST poses stiffer generalization test than MNIST","New hand-written Kannada digits challenge MNIST-style CNNs","Kannada digit set: MNIST drop-in with harder writer splits","CNN baseline drops to 76% on Kannada-MNIST out-of-domain","Kannada-MNIST: tougher test for handwriting recognition models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the worst-cohort selection rule—ranking volunteers by how often a MNIST-trained CNN maps their Kannada 0, 3, and 7 to the similar-looking MNIST classes 0 and 2—ranks true out-of-sample generalization difficulty, so the 96.85% figure describes what a classifier should expect on a new writer.","fun_headline_variants_meta":{"raw":{"variants":["Kannada-MNIST poses stiffer generalization test than MNIST","New hand-written Kannada digits challenge MNIST-style CNNs","Kannada digit set: MNIST drop-in with harder writer splits","CNN baseline drops to 76% on Kannada-MNIST out-of-domain","Kannada-MNIST: tougher test for handwriting recognition models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000613,"raw_usage":{"total_tokens":2838,"prompt_tokens":919,"completion_tokens":1919,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":1824}},"tokens_in":535,"tokens_out":1919,"duration_ms":13498,"temperature":1.0,"reasoning_tokens":1824,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T15:17:50.522783+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a CNN on 57 of the 65 volunteers and test on a randomly chosen eighth volunteer's digits instead of the worst-scoring cohort; if accuracy lands near the above-99% figure the paper saw in random splits, the 96.85% headline is an artifact of the selection rule. Inspecting per-writer accuracy inside the worst cohort would also show whether the difficulty is spread across all eight writers or driven by one or two.","supporting_citations":[{"cited_title":"Mnist handwritten digit database","cited_arxiv_id":null,"evidence_quote":"It is the standard MNIST handwritten digit benchmark that Kannada-MNIST is designed to replace, supplying the 28x28 format and the comparison baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It is the current largest Kannada character dataset used in academic literature, with only 25 samples per numeral, establishing the paucity this paper addresses."},{"cited_title":"Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms","cited_arxiv_id":null,"evidence_quote":"It is a prior drop-in replacement for MNIST, establishing the format and use-case that Kannada-MNIST follows."},{"cited_title":"Deep learning for classical japanese literature","cited_arxiv_id":null,"evidence_quote":"It is a prior MNIST-format digit benchmark in a non-Latin script, used as a comparator for the generalization challenge."},{"cited_title":"https://gist.github.com/mineshpatel1/ 209038c64c19d5e78e0a878320797631#file-sudoku_cv-py, 2017","cited_arxiv_id":null,"evidence_quote":"It supplies the grid-image preprocessing and slicing pipeline used to extract individual digits from scanned sheets."},{"cited_title":"https://medium.com/@o.kroeger/ tensorflow-mnist-and-your-own-handwritten-digits-4d1cd32bbab4 , 2016","cited_arxiv_id":null,"evidence_quote":"It supplies the MNIST-ization procedure: thresholding, padding, and center-of-mass shifting used to produce 28x28 images."},{"cited_title":"Fonts-2-Handwriting: A Seed-Augment-Train framework for universal digit classification","cited_arxiv_id":"1905.08633","evidence_quote":"It is the prior synthetic-to-real Kannada digit framework that supplies the font-based seed generation approach and the 1280-digit real-world test set used as a third evaluation set."}],"review_version":1}