Label Errors in the Tobacco3482 Dataset

Gordon Lim; Kevin Leach; Stefan Larson

REVIEW 4 major objections 5 minor 19 references

Label Errors in the Tobacco3482 Dataset

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that 11.7% of Tobacco3482 documents are improperly annotated—151 with no valid category, 258 mislabeled—and that 583 documents (16.7%) have multiple valid labels; re-scoring a strong model's errors with that knowledge…

desk verdict A genuinely useful first audit of Tobacco3482 that gets the direction right; the exact percentages need a reliability caveat because they rest on one annotator's judgment. read the letter →

arxiv 2412.13140 v1 pith:KN6T2VMW submitted 2024-12-17 cs.CV

Gordon Lim , Stefan Larson , Kevin Leach This is my paper · ORCID

classification cs.CV

keywords Tobacco3482labelerrorsdocumentimageclassificationannotationqualitymulti-labelimagesbenchmarkevaluationmodelmistakes

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that Tobacco3482, a standard benchmark for document image classification, is substantially noisier than its users assume. A complete manual re-annotation of all 3,482 images, guided by category definitions the authors wrote down first, yields 151 documents with no valid category, 258 documents assigned to the wrong category, and 583 documents with more than one valid label. The authors then use their re-annotation to re-score the mistakes of a top-performing DiT document transformer: 196 of the model's 554 errors are valid alternative labels or uncategorizable documents, so its accuracy rises from 84.1% to 89.7% when those are counted as correct. If the audit is right, published accuracy figures on Tobacco3482 understate model capability and should not be read as clean measures of classification skill.

What carries the argument

The central object is a re-annotation protocol with three error types: unknown, mislabeled, and multiple labels. The paper first fixes explicit definitions for each of the ten Tobacco3482 categories (for example, a Letter must have physical addressing and a sign-off, while a Memo is addressed with To/From), then applies those definitions to all 3,482 images and records every case where the original single label fails. This protocol is what generates the 151/258/583 counts and the overlap structure between categories, and it is also how the authors convert a model's predicted labels into a verdict of 'valid alternative label' or 'no valid label' when re-scoring the DiT model's mistakes.

What would settle it

Have a second annotator, blinded to the original labels and working only from the paper's published category guidelines, re-annotate all 3,482 images independently; if the counts of unknown, mislabeled, and multi-label documents diverge materially from 151, 258, and 583—say, by more than a few percentage points—the paper's headline error rates and its '35% of mistakes' attribution are not stable.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the label layer of Tobacco3482 is not a single ground truth. The authors inspect every image and report 151 unknown-label cases, 258 wrong-label cases, and 583 multi-label cases, with errors concentrated in categories such as Letter (about 24% mislabeled, mostly documents they would call Memos) and Scientific (about 24% mislabeled and 14.6% unknown). To show the practical effect, they evaluate a DiT model with four-fold cross-validation on the original labels, collect its 554 errors, and find that 147 are predictions of a valid alternative label and 49 are predictions for documents that have no valid label at all. Re-scoring those 196 cases as correct moves the model from 84.1% to 89.7% top-1 accuracy.

Load-bearing premise

The load-bearing premise is that the paper's self-authored labeling guidelines and the single annotator's application of them are the correct way to decide what each Tobacco3482 document really is, and the paper reports no second annotation or adjudication to test that judgment.

Editorial extensions

If this is right

Reported top-1 accuracy on Tobacco3482 understates model capability; the paper's own DiT model jumps from 84.1% to 89.7% once valid alternative labels and unknown cases are counted as correct.
Benchmark comparisons on the raw dataset can mis-rank models, since a large share of measured errors are annotation artifacts rather than classification failures.
Future evaluations should report multi-label-aware and unknown-aware metrics instead of single-label accuracy on the original labels.
The 583 multi-label documents imply that multi-label classification is a more faithful task for this benchmark than the original single-label task.
Error rates are concentrated in certain categories (Letter, Scientific), so per-category accuracy numbers on the original labels are especially unreliable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the same single-annotator audit were repeated on other document benchmarks drawn from the same source collection, similarly high label-error rates might appear, so reported gains on those benchmarks should be read with caution.
A corrected, multi-label version of Tobacco3482 could serve as a testbed for deciding whether improvements in published accuracy come from better models or from better fit to noisy labels.
Because 49 of the model's 'mistakes' are on documents with no valid category, a practical classifier deployed on such documents would need a rejection or 'other' option; without one, accuracy metrics are unfairly harsh.
One testable implication: a model trained or evaluated with multi-label structure (for example, predicting both Report and Memo) should beat a single-label model on the paper's re-annotated labels even if they tie on the original labels.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First useful audit of Tobacco3482. The headline numbers—11.7% unknown/mislabeled, 16.7% multi-label, 35% of DiT mistakes not really mistakes—are plausible and internally consistent, and the release of annotations and code is a genuine contribution to a subfield that has been flying blind. The independent check via TTID's original multi-label annotations (3.8% of documents carry multiple category labels there) gives some external support for the multi-label finding, which is more than most audits offer.

The soft spots are real but not fatal: the entire count rests on one annotator applying the authors' own Table 1 guidelines, with no inter-annotator agreement, no adjudication, and no error bars. Some guidelines are explicitly subjective (e.g., Letter excludes memos using 'letter' in the subject; ADVE excludes magazine covers). A second reasonable annotator could shift the numbers several points. The paper should present these as estimates, not ground truth, and ideally run a second annotator on a sample to bound the uncertainty. The model-impact analysis inherits this fragility, so the 89.7% corrected accuracy should also be labeled as a single-annotator estimate.

That said, the internal arithmetic checks out (151+258=409=11.7%; 196/554=35.4%), the code and annotations are inspectable, and the qualitative conclusion—that Tobacco3482 has meaningful label noise—is well-supported by the examples and the TTID cross-check. The paper is short and honest about being a first pass.

Who should read it: anyone using Tobacco3482 or RVL-CDIP for evaluation, and people working on dataset auditing methodology. It deserves a serious referee. I'd recommend acceptance with a mandatory revision that adds a reliability analysis or at minimum careful caveats, and possibly a second annotator on a subset to quantify agreement.

Referee Report

4 major / 5 minor

Summary. The paper reports a manual audit of all 3,482 images in the Tobacco3482 document classification dataset. The authors first define a set of category guidelines (Table 1), then re-annotate every document, finding 151 samples with no valid Tobacco3482 category, 258 samples with a wrong original label, and 583 samples with multiple valid labels. They further analyze the mistakes of a DiT model evaluated under 4-fold cross-validation, reporting 554 total mistakes, of which 196 are claimed to be valid alternative labels or unlabeled/unknown cases, raising the model's accuracy from 84.1% to 89.7% under their reannotation criteria.

Significance. If the audit is reliable, this is a useful contribution to the growing literature on benchmark label quality, providing the first systematic error count for Tobacco3482 and extending the authors' earlier RVL-CDIP analysis. The paper is transparent about its annotation artifacts, releasing the new labels and code, and its headline numbers are internally consistent (409/3482 = 11.7%, 583/3482 = 16.7%, 196/554 = 35.4%). The main value is conditional on the reproducibility of the annotation procedure: all quantitative claims rest on a single annotator's application of self-authored guidelines, with no inter-annotator agreement or external adjudication. With an additional reliability study or a substantially weakened framing, the paper could be a valuable dataset-quality reference; in its current form, the central percentages are plausible but not established as stable ground truth.

major comments (4)

[§2.2, Table 1] The quantitative claims (151 unknown, 258 mislabeled, 583 multi-label) are derived from a single annotator applying guidelines that contain explicitly subjective boundary decisions, for example 'Letter excludes memos using letter in its subject', 'Note excludes Reports with Note in title', and 'ADVE excludes magazine covers and content drafts'. The paper reports no inter-annotator agreement, no independent adjudication, and no estimate of annotation variability. Because every downstream number—11.7%, 16.7%, and the 196/554 correction—inherits this single annotator's judgment, this is a load-bearing methodological gap. Please add an inter-annotator agreement study on a sample (reporting, for example, Cohen's kappa for unknown, mislabeled, and multi-label decisions) with adjudication of disagreements, or alternatively reframe all counts as one annotator's assessment and add a prominent limitations paragraph that removes the implication of ground truth.
[§3] The '89.7% accuracy' is computed by post-hoc reclassifying 196 of the model's predictions as correct; it is not a model trained or evaluated on a relabeled version of Tobacco3482. The subsequent comparison with DocXclassifier's 90.7% accuracy in [16] is therefore not a controlled comparison, since the latter method was trained and evaluated under its own protocol on the original labels. Please state explicitly that 89.7% is a counterfactual reannotation upper bound, not a measured accuracy from a retrained model, and soften the claim that the corrected score 'brings it closer to' a state-of-the-art method.
[§2.2] The three problematic label types (unknown, mis-labeled, multi-label) are presented as separate counts, but the paper never states whether these categories are mutually exclusive. In particular, a document whose original label is wrong can have two or more valid alternative labels, making it both 'mis-labeled' and 'multi-label'. As written, the reader cannot compute the unique number of documents with any label issue, and the sum 151 + 258 + 583 = 992 may overstate the total number of affected samples. Please report the pairwise and triple intersections among the three categories and give a unified rate for documents with at least one annotation problem.
[§2.1] The TTID cross-check identifies 134 of 1,707 located documents (roughly 7.8%) as having multiple TTID category annotations, yet the authors' own review finds 583 multi-label samples (16.7% of the dataset). The paper does not reconcile this large discrepancy or use the TTID annotations as an independent partial check on the authors' multi-label counts. A short discussion of whether the difference reflects intentionally narrow TTID annotations, different category schemas, or annotator interpretation would substantially strengthen confidence in the audit.

minor comments (5)

[§1] The sentence 'our findings highlight flaws in the Tobacco3482 dataset concerining data quality' contains a typo: 'concerining' should be 'concerning'.
[§3] The same paragraph reports '84.1% top-1 accuracy' on one line and 'the original 84.0% accuracy' two sentences later; please make the reported baseline accuracy consistent.
[§2.2] The subsection heading 'Multiple Lables' should be 'Multiple Labels'.
[Figure 1] The figure caption begins 'un-problematicproblematic', which appears to be a formatting error and should read 'un-problematic and problematic'.
[§3] For reproducibility, please report the DiT fine-tuning setup used in the 4-fold cross-validation (e.g., whether folds were stratified, number of epochs, learning rate, batch size, and random seeds).

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the audit is an empirical annotation exercise; the one same-team citation is procedural, not load-bearing.

full rationale

The paper's central numbers—151 unknown, 258 mislabeled, 583 multi-label, and the resulting 196 of 554 DiT mistakes—are outputs of a direct manual re-annotation of all 3,482 images under the authors' Table 1 rubric. They are not derived from one another, nor are they implied by a fitted parameter or by a mathematical identity. The downstream model analysis is a set comparison between independently produced DiT predictions and the annotation set: 147 predictions matched a multi-label annotation and 49 matched an 'unknown' designation, which is a logical consequence of the annotations, not a circular derivation. The only same-team citation ([7]) motivates the annotation procedure but supplies no Tobacco3482 counts and is not invoked as an authoritative theorem. Concerns about single-annotator reliability and lack of inter-annotator agreement are validity and reproducibility limitations, not circularity, and per the review rules they do not constitute a demonstrated circular step. The self-citation is therefore minor and non-load-bearing.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central measurement depends on the authors' self-defined labeling guidelines; there are no fitted numeric parameters, but the guidelines and the multi-label evaluation rule are assumptions adopted for this paper.

assumptions (3)

ad hoc to paper The authors' Table 1 guidelines correctly define the 10 Tobacco3482 categories.
Section 2.2: since no official guidelines exist, the authors create these rules; all error counts are measured against them.
domain assumption A document may have multiple valid labels, and a model prediction matching any one of them should count as correct in the revised evaluation.
Section 3: the multi-label annotations are used to reclassify 147 model predictions as correct; this changes the evaluation protocol from single-label to multi-label matching.
domain assumption The TTID manual category annotations, where matched for 1,707 of 3,482 documents, are a reliable signal that multiple labels exist.
Section 2.1: 134 matched documents have multiple TTID category annotations, cited as evidence that some Tobacco3482 documents have valid alternate labels.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Label Errors in the Tobacco3482 Dataset." pith.science (2026). https://pith.science/paper/KN6T2VMW

@misc{pith2026241213140,
  author       = {Pith},
  title        = {Pith review of: Label Errors in the Tobacco3482 Dataset},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KN6T2VMW}},
  note         = {Machine review of arXiv:2412.13140}
}

read the original abstract

Tobacco3482 is a widely used document classification benchmark dataset. However, our manual inspection of the entire dataset uncovers widespread ontological issues, especially large amounts of annotation label problems in the dataset. We establish data label guidelines and find that 11.7% of the dataset is improperly annotated and should either have an unknown label or a corrected label, and 16.7% of samples in the dataset have multiple valid labels. We then analyze the mistakes of a top-performing model and find that 35% of the model's mistakes can be directly attributed to these label issues, highlighting the inherent problems with using a noisily labeled dataset as a benchmark. Supplementary material, including dataset annotations and code, is available at https://github.com/gordon-lim/tobacco3482-mistakes/.

Figures

Figures reproduced from arXiv: 2412.13140 by the authors.

**Figure 2.** Examples of problematic samples from Tobacco3482. Top row: documents where the valid label is unknown. Middle row: [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** UpSet plot [10] of multi-label Tobacco3482 category annotations. A majority of the documents with multiple labels are documents that are both Reports and Memos. 3. Impact of Label Issues on the Classification Task To contextualize the impact of these data labeling issues on model evaluation, we analyze the mistakes of a DiT model [11]. We used a DiT model from Hugging Face4 , using RVL-CDIP [4] pre-trained weights… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 19 canonical work pages

[16]

DocXclassifier: towards a robust and interpretable deep neu- ral network for document image classification

Saifullah, Stefan Agne, Andreas Dengel, and Sheraz Ahmed. DocXclassifier: towards a robust and interpretable deep neu- ral network for document image classification. International Journal on Document Analysis and Recognition (IJDAR) , 27(3), 2024. 1, 4

work page 2024
[1]

Two stream deep network for docu- ment image classification

Muhammad Nabeel Asim, Muhammad Usman Ghani Khan, Muhammad Imran Malik, Khizar Razzaque, Andreas Den- gel, and Sheraz Ahmed. Two stream deep network for docu- ment image classification. In Proceedings of the 2019 Inter- national Conference on Document Analysis and Recognition (ICDAR), 2019. 1

work page 2019
[2]

Do we train on test data? purging CIFAR of near-duplicates.Journal of Imaging, 6(6),

Bj ¨orn Barz and Joachim Denzler. Do we train on test data? purging CIFAR of near-duplicates.Journal of Imaging, 6(6),

work page
[3]

ImageNet: A large-scale hierarchical im- age database

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical im- age database. In Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009. 1

work page 2009
[4]

Harley, Alex Ufkes, and Konstantinos G

Adam W. Harley, Alex Ufkes, and Konstantinos G. Derpa- nis. Evaluation of deep convolutional nets for document im- age classification and retrieval. In Proceedings of the Inter- national Conference on Document Analysis and Recognition (ICDAR), 2015. 1, 4

work page 2015
[5]

Document image classification using SqueezeNet convolutional neural network

Mohammad Hassanpour and Hamed Malek. Document image classification using SqueezeNet convolutional neural network. In Proceedings of the 5th Iranian Conference on Signal Processing and Intelligent Systems (ICSPIS), 2019. 1

work page 2019
[6]

Britto, Rodrigo Minetto, and David Menotti

Rayson Laroca, Valter Estevam, Alceu S. Britto, Rodrigo Minetto, and David Menotti. Do we train on test data? the impact of near-duplicates on license plate recognition. In Proceedings of the 2023 International Joint Conference on Neural Networks (IJCNN), 2023. 1

work page 2023
[7]

On evaluation of document classification with RVL-CDIP

Stefan Larson, Gordon Lim, and Kevin Leach. On evaluation of document classification with RVL-CDIP. In Proceedings of the 17th Conference of the European Chapter of the As- sociation for Computational Linguistics (EACL), 2023. 1, 2, 4

work page 2023

Show all 19 references

[8]

De-identification of sensitive personal data in datasets de- rived from IIT-CDIP

Stefan Larson, Nicole Cornehl Lima, Santiago Pedroza Diaz, Amogh Manoj Joshi, Siddharth Betala, Jamiu Tunde Suleiman, Yash Mathur, Kaushal Kumar Prajapati, Ramla Alakraa, Junjie Shen, Temi Okotore, and Kevin Leach. De-identification of sensitive personal data in datasets de- r...

2024
[9]

Grossman, and Jefferson Heard

David Lewis, Gady Agam, Shlomo Argamon, Ophir Frieder, D. Grossman, and Jefferson Heard. Building a test collection for complex document information processing. In Proceed- ings of the 29th Annual International ACM SIGIR Confer- ence on Research and Development in Information ...
[10]

UpSet: Visualization of intersecting sets

Alexander Lex, Nils Gehlenborg, Hendrik Strobelt, Romain Vuillemot, and Hanspeter Pfister. UpSet: Visualization of intersecting sets. IEEE Transactions on Visualization and Computer Graphics, 20(12):1983–1992, 2014. 4

1983
[11]

DiT: Self-supervised pre-training for docu- ment image transformer

Junlong Li, Yiheng Xu, Tengchao Lv, Lei Cui, Cha Zhang, and Furu Wei. DiT: Self-supervised pre-training for docu- ment image transformer. In Proceedings of the 30th ACM International Conference on Multimedia, 2022. 4

2022
[12]

The nine lives of Ima- geNet: A sociotechnical retrospective of a foundation dataset and the limits of automated essentialism

Sasha Luccioni and Kate Crawford. The nine lives of Ima- geNet: A sociotechnical retrospective of a foundation dataset and the limits of automated essentialism. Journal of Data- centric Machine Learning Research, 2024. 1

2024
[13]

Embedded textual content for document image classification with convolutional neural networks

Lucia Noce, Ignazio Gallo, Alessandro Zamberletti, and Alessandro Calefati. Embedded textual content for document image classification with convolutional neural networks. In Proceedings of the 2016 ACM Symposium on Document En- gineering, 2016. 1

2016
[14]

Confident learning: Estimating uncertainty in dataset labels

Curtis Northcutt, Lu Jiang, and Isaac Chuang. Confident learning: Estimating uncertainty in dataset labels. J. Artif. Int. Res., 70:1373–1411, May 2021. 1

2021
[15]

Northcutt, Anish Athalye, and Jonas Mueller

Curtis G. Northcutt, Anish Athalye, and Jonas Mueller. Per- vasive label errors in test sets destabilize machine learning benchmarks. In Proceedings of the 35th Conference on Neu- ral Information Processing Systems Track on Datasets and Benchmarks, 2021. 1

2021
[17]

The reality of high performing deep learning models: A case study on document image classification

Saifullah, Stefan Agne, Andreas Dengel, and Sheraz Ahmed. The reality of high performing deep learning models: A case study on document image classification. IEEE Access, 12:103537–103564, 2024. 4

2024
[18]

Label errors in BANK- ING77

Cecilia Ying and Stephen Thomas. Label errors in BANK- ING77. In Proceedings of the Third Workshop on Insights from Negative Results in NLP, 2022. 1

2022
[19]

Automatic document logo detection

Guangyu Zhu and David Doermann. Automatic document logo detection. In Proceedings of the 9th International Con- ference on Document Analysis and Recognition (ICDAR) ,

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [16]

DocXclassifier: towards a robust and interpretable deep neu- ral network for document image classification

Saifullah, Stefan Agne, Andreas Dengel, and Sheraz Ahmed. DocXclassifier: towards a robust and interpretable deep neu- ral network for document image classification. International Journal on Document Analysis and Recognition (IJDAR) , 27(3), 2024. 1, 4

work page 2024

[2] [1]

Two stream deep network for docu- ment image classification

Muhammad Nabeel Asim, Muhammad Usman Ghani Khan, Muhammad Imran Malik, Khizar Razzaque, Andreas Den- gel, and Sheraz Ahmed. Two stream deep network for docu- ment image classification. In Proceedings of the 2019 Inter- national Conference on Document Analysis and Recognition (ICDAR), 2019. 1

work page 2019

[3] [2]

Do we train on test data? purging CIFAR of near-duplicates.Journal of Imaging, 6(6),

Bj ¨orn Barz and Joachim Denzler. Do we train on test data? purging CIFAR of near-duplicates.Journal of Imaging, 6(6),

work page

[4] [3]

ImageNet: A large-scale hierarchical im- age database

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical im- age database. In Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009. 1

work page 2009

[5] [4]

Harley, Alex Ufkes, and Konstantinos G

Adam W. Harley, Alex Ufkes, and Konstantinos G. Derpa- nis. Evaluation of deep convolutional nets for document im- age classification and retrieval. In Proceedings of the Inter- national Conference on Document Analysis and Recognition (ICDAR), 2015. 1, 4

work page 2015

[6] [5]

Document image classification using SqueezeNet convolutional neural network

Mohammad Hassanpour and Hamed Malek. Document image classification using SqueezeNet convolutional neural network. In Proceedings of the 5th Iranian Conference on Signal Processing and Intelligent Systems (ICSPIS), 2019. 1

work page 2019

[7] [6]

Britto, Rodrigo Minetto, and David Menotti

Rayson Laroca, Valter Estevam, Alceu S. Britto, Rodrigo Minetto, and David Menotti. Do we train on test data? the impact of near-duplicates on license plate recognition. In Proceedings of the 2023 International Joint Conference on Neural Networks (IJCNN), 2023. 1

work page 2023

[8] [7]

On evaluation of document classification with RVL-CDIP

Stefan Larson, Gordon Lim, and Kevin Leach. On evaluation of document classification with RVL-CDIP. In Proceedings of the 17th Conference of the European Chapter of the As- sociation for Computational Linguistics (EACL), 2023. 1, 2, 4

work page 2023