Pith. sign in

REVIEW 3 major objections 5 minor 21 references

Zero-Shot Prompting and Few-Shot Fine-Tuning: Revisiting Document Image Classification Using Large Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper establishes that large language models can classify scanned documents with almost no training data: generative fine-tuning of Mistral-7B on ten samples per class reaches 72.5% accuracy on RVL-CDIP, and 100 samples per class…

desk verdict A genuinely useful few-shot document classification benchmark whose headline comparison to fully trained BERT on RVL-CDIP-40K is not apples-to-apples because the test sets and OCR quality change together. read the letter →

arxiv 2412.13859 v1 pith:UQ6OE76C submitted 2024-12-18 cs.CV

Anna Scius-Bertrand , Michael Jungo , Lars Vögtlin , Jean-Marc Spat , Andreas Fischer This is my paper · ORCID
classification cs.CV
keywords DocumentImageClassificationOCRLargeLanguageModelsFew-ShotLearningZero-ShotPromptingFine-TuningLoRARVL-CDIP
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether document image classification can be done with almost no labeled training data, using large language models that read OCR text. It reports a benchmark on RVL-CDIP showing that a zero-shot multimodal model reaches 69.9% accuracy, and that generative fine-tuning of Mistral-7B with only ten samples per class reaches 72.5% accuracy on a five-fold 160-sample test selection. With 100 samples per class, classifier fine-tuning reaches 83.4%, close to the 85.0% reported for a fully trained BERT model using 320,000 samples. The paper's point is that LLMs collapse the data requirement for document classification by orders of magnitude, while leaving room for improvement.

What carries the argument

The argument rests on a purpose-built evaluation scenario, RVL-CDIP-160x5, consisting of five disjoint 160-sample subsets (ten per class) drawn from the RVL-CDIP test set, with fresh high-quality OCR from Amazon Textract. The fine-tuning results are produced by quantized LoRA adaptation of Mistral-7B (4-bit base weights, rank-8 adapters) in two modes: generative, where the model predicts a class token appended after a 'Classification:' directive, and classifier, where a classification head replaces the language-model head. Zero-shot prompting uses a system prompt refined with GPT-4, evaluated on OCR text or document images (GPT-4-Vision). The RVL-CDIP-40K test set with older OCR is retained as a reference to check how representative the small subsets are.

What would settle it

Take the 1,600-sample fine-tuned classifier and generative model, run them on the full RVL-CDIP-40K test set using the same high-quality OCR used for the small test sets, and compare accuracy on the 40K set to the 83.4% and 72.5% figures; a large drop would falsify the representativeness assumption. Alternatively, compute accuracy separately for documents with low OCR confidence to test whether the small subsets skew toward legible documents.

Watch

Extended reading notes

Core claim

The central empirical discovery is that generative fine-tuning of a small open-source LLM, Mistral-7B, with a LoRA adapter on OCR text is extraordinarily sample-efficient for document classification. Using only 160 training documents (ten per class), it reaches 72.5% mean accuracy on the RVL-CDIP-160x5 test sets, outperforming fully fine-tuned RoBERTa (59.8%), KNN embeddings, and image-only Donut. With 1,600 samples (100 per class), a classifier head on the same base model reaches 83.4%, approaching the 85.0% accuracy of a fully trained BERT model on the full RVL-CDIP-40K test set. In zero-shot prompting, GPT-4-Vision, which sees only document images, reaches 69.9% without any training samples. The comparison establishes that both zero-shot prompting and few-shot fine-tuning are viable regimes for document classification.

Load-bearing premise

The headline accuracy numbers rest on the assumption that the five randomly chosen 160-document test subsets fairly represent the full RVL-CDIP test set, so that results on them transfer to real-world document streams.

Editorial extensions

If this is right

  • If the results hold, document classification pipelines can be built with tens to hundreds of labeled documents instead of hundreds of thousands.
  • Zero-shot prompting with a multimodal LLM can serve as an immediate baseline for new document categories before any annotation effort.
  • Generative fine-tuning is preferable to classifier fine-tuning when training data are very scarce, because it retains output flexibility and is more stable across sample sizes.
  • The 83.4% result with 100 samples per class implies that LLM-based classifiers can rival fully trained specialized models on RVL-CDIP, despite being an order of magnitude larger in parameters.
  • OCR quality is a key variable: the gap between RVL-CDIP-160x5 and RVL-CDIP-40K results for OCR-based models shows that high-quality OCR is needed to realize these accuracies.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The representativeness of the RVL-CDIP-160x5 subsets is the main risk: if the five 160-sample draws are easier than the full test set, the 72.5% and 83.4% figures overstate real-world accuracy. A direct evaluation on the full 40K test set with matched high-quality OCR would settle this.
  • Because the generative model was trained only on OCR text, it ignores layout and image cues; combining the LoRA-tuned text model with a visual encoder might push few-shot accuracy closer to the 97% reported for fully trained multimodal systems.
  • The benchmark design could be extended to out-of-distribution document types to test whether the few-shot gains transfer to categories absent from pretraining, rather than only the 16 RVL-CDIP classes.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper benchmarks zero-shot prompting and few-shot fine-tuning for document image classification on RVL-CDIP using a range of models: generative LLMs (Mistral-7B, Mixtral, GPT-3.5, GPT-4, GPT-4-Vision), text embeddings with KNN, fine-tuned RoBERTa, LoRA fine-tuned Mistral-7B in classifier and generative modes, and the image-based Donut model. The authors introduce RVL-CDIP-160x5, five random 160-sample test subsets with Amazon Textract OCR, and report mean accuracy and standard deviation over the subsets. Their headline findings are that generative LoRA fine-tuning of Mistral-7B reaches 72.5% accuracy with ten samples per class, classifier fine-tuning reaches 83.4% with 100 samples per class, and they compare this with an 85.0% fully trained BERT result on RVL-CDIP-40K from the literature. They conclude that zero-shot and few-shot LLM-based document classification is feasible and that generative fine-tuning is sample-efficient.

Significance. If the claims hold, the paper provides a useful benchmark for few-shot document classification and a careful set of comparisons across prompting, embedding, and fine-tuning approaches. The experimental design has notable strengths: five random test subsets with mean and standard deviation, validation-based selection of prompts and hyperparameters, and inclusion of both the custom 160x5 test sets and the full 40K test set as a reference. The headline comparison with BERT is striking and would be of broad interest, but as reported it is confounded by simultaneous changes in test subset and OCR quality. The internal comparisons on RVL-CDIP-160x5 appear sound; the cross-benchmark contextual claim needs additional support.

major comments (3)
  1. [Section 4.4, Table 6] The central contextual claim that 83.4% few-shot Mistral-7B-Class accuracy is 'a notable achievement when compared to the 85.0% accuracy reported in [2]' is confounded by two factors that change simultaneously: the test set and the OCR engine. RVL-CDIP-160x5 uses Amazon Textract OCR, while RVL-CDIP-40K uses the original IIT-CDIP OCR, which the paper itself describes in Section 4.1 as 'unquestionably' lower quality. The same Mistral-7B-Class checkpoint scores only 66.6% on RVL-CDIP-40K (Table 5), and the paper attributes the 16.8-point gap to OCR quality, but this attribution is not tested directly. A matched-OCR evaluation, for example running Textract on a random subset of the 40K test set or running the original OCR on the 160x5 documents, is needed before the '83.4% vs 85.0%' comparison can support the stated conclusion. As written, the comparison is not established.
  2. [Section 4.4] The conclusion that 'the five-fold selection of 160 test samples is, indeed, representative' is supported only by the OCR-free Donut control. Donut's similar accuracy on RVL-CDIP-40K and RVL-CDIP-160x5 controls for image-content bias, but it does not control for the OCR-quality shift that affects all OCR-based models in Table 5. The systematic drop observed for RoBERTa and Mistral-7B on RVL-CDIP-40K may indeed be due to inferior OCR, but it could also be caused by a subset bias that interacts with text-based methods. The representativeness claim should either be restricted to image-based models or be backed by an OCR-aware check on the same documents under both OCR conditions.
  3. [Section 4.1, Table 5] The RVL-CDIP-40K results are reported as single-run values without standard deviations or repetition, even though they are used to support the OCR-quality hypothesis and serve as the reference for the headline comparison with BERT. Since the 160x5 results all carry mean and standard deviation, the asymmetry in reporting makes it difficult to assess the reliability of the 40K numbers. Please report uncertainty across repeated runs, or at minimum explicitly label these values as exploratory single-run references and avoid drawing strong conclusions from small differences on this set.
minor comments (5)
  1. [Section 4.1] There is a typo in 'specifying a temparature of0': 'temperature' is misspelled and a space is missing before the zero.
  2. [Section 4.5, Table 6] The sentence introducing Table 6 says the 160x5 results are 'put into context' with literature results on RVL-CDIP-40K; it should explicitly state that this context comparison crosses both test-set definitions and OCR sources, since that is exactly the caveat raised in the major comments.
  3. [References] References [8] and [9] both point to the same Mistral 7B paper (arXiv:2310.06825); one duplicate should be removed and the in-text citations merged.
  4. [Figure 3] The t-SNE plot would be more reproducible if the parameters (perplexity, number of iterations, initialization) were reported, or if the figure is intended as purely illustrative this should be stated.
  5. [Table 3] Several models produce high invalid-answer rates; a sentence explicitly stating that invalid answers are counted as incorrect (rather than excluded from the accuracy computation) would help readers interpret the reported accuracy numbers.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the headline results are direct measurements on held-out test sets, and the representativeness control uses an independent model.

full rationale

This paper is an empirical benchmark, not a derivation. The central reported numbers, such as 72.5% for Mistral-7B-Gen at 160 training samples and 83.4% for Mistral-7B-Class at 1600 samples, are direct accuracy measurements on the held-out RVL-CDIP-160x5 test sets. They are not obtained by fitting a parameter to a target quantity and then re-predicting that same quantity. The choice of prompt P2 on a small validation set and the selection of LoRA rank and alpha on validation are standard model-selection steps; the reported test accuracies come from disjoint test sets and are not defined by those choices. The representativeness claim for the 160x5 subsets is supported by an independent OCR-free model, Donut, whose accuracy is similar on the 160x5 subsets and the full 40K test set. This is an empirical control rather than a definitional equivalence. No self-citations appear, no uniqueness theorem is invoked, and no known result is merely renamed. The cross-set comparison to the fully trained BERT result (85.0%) is confounded by OCR quality and test-set composition, but that is a validity or correctness concern, not a circularity. No step in the paper reduces by construction to its own input.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim relies on empirical measurements rather than derivation. The main imported assumptions are dataset label correctness, OCR adequacy, and the representativeness of the custom test subset. Hyperparameters selected on the validation set are standard model selection but are listed as free parameters for transparency.

free parameters (3)
  • KNN k and distance metric = not reported; selected from k in {1,3,5,7,9} and Euclidean or cosine
    Section 4.1 states the k and metric are optimized with respect to classification accuracy on the validation set, but the selected values are not reported.
  • LoRA rank r and alpha = r=8, alpha=16
    Section 4.1 says these values were chosen based on preliminary experiments on the validation set, so they are tuned rather than fixed by theory.
  • System prompt P2 = P2, the GPT-4-enhanced prompt
    Section 4.2 reports that P2 was selected over P1 and P3 using GPT-3.5 validation accuracy on 32 training samples, making prompt choice a validation-tuned component.
assumptions (4)
  • domain assumption Amazon Textract OCR on the sampled RVL-CDIP subsets preserves enough text content and reading order for classification.
    Section 3.1 says Textract performed reasonably well, and Section 2 acknowledges OCR imperfections; the method depends on this assumption.
  • domain assumption RVL-CDIP ground-truth class labels are correct.
    Labels are used throughout as the evaluation reference without independent verification; introduced in Section 2.
  • domain assumption RVL-CDIP-160x5 is representative of the full RVL-CDIP-40K test set.
    Section 4.4 concludes representativeness from Donut agreement between the two test scenarios, but OCR-based models show large accuracy gaps between them.
  • domain assumption The accuracy gap between 160x5 and 40K for OCR-based models is caused by lower-quality IIT-CDIP OCR rather than test set bias.
    Section 4.4 states this as the likely explanation; it is a hypothesis, not a directly controlled measurement.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Zero-Shot Prompting and Few-Shot Fine-Tuning: Revisiting Document Image Classification Using Large Language Models." pith.science (2026). https://pith.science/paper/UQ6OE76C

@misc{pith2026241213859,
  author       = {Pith},
  title        = {Pith review of: Zero-Shot Prompting and Few-Shot Fine-Tuning: Revisiting Document Image Classification Using Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UQ6OE76C}},
  note         = {Machine review of arXiv:2412.13859}
}
read the original abstract

Classifying scanned documents is a challenging problem that involves image, layout, and text analysis for document understanding. Nevertheless, for certain benchmark datasets, notably RVL-CDIP, the state of the art is closing in to near-perfect performance when considering hundreds of thousands of training samples. With the advent of large language models (LLMs), which are excellent few-shot learners, the question arises to what extent the document classification problem can be addressed with only a few training samples, or even none at all. In this paper, we investigate this question in the context of zero-shot prompting and few-shot model fine-tuning, with the aim of reducing the need for human-annotated training samples as much as possible.

Figures

Figures reproduced from arXiv: 2412.13859 by the authors.

Figure 1
Figure 1. Example document images from the RVL-CDIP dataset. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Three system prompts considered for document classification. Note that [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Embedding of the 1 600 training samples using the OpenAI-large model, visualized with t-SNE. training samples or more are used. The results also indicate that almost no invalid responses are generated by the fine-tuned models. One model stands out with an excellent performance for few-shot learning: Generative fine-tuning of Mistral-7B. Even when providing only 160 training samples (10 samples per class), the LLM ac… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 12 canonical work pages

  1. [2]

    Visual and textual deep feature fusion for document image classification

    Bakkali, S., Ming, Z., Coustaty, M., and Rusiñol, M. Visual and textual deep feature fusion for document image classification. InProc. Int. Conf. on Com- puter Vision and Pattern Recognition Workshops (CVPR W) (2020), pp. 562–563

    work page 2020

  2. [1]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agar w al, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  3. [3]

    D., Dhariw al, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariw al, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems (NeurIPS) 33 (2020), 1877–1901

    work page 2020

  4. [4]

    Qlora: Efficient finetuning of quantized llms.Advances in Neural Information Processing Systems 36 (2024)

    Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. Qlora: Efficient finetuning of quantized llms.Advances in Neural Information Processing Systems 36 (2024)

    work page 2024

  5. [5]

    K., Guzman, S., Mastrapas, G., Sturua, S., W ang, B., et al

    Günther, M., Ong, J., Mohr, I., Abdessalem, A., Abel, T., Akram, M. K., Guzman, S., Mastrapas, G., Sturua, S., W ang, B., et al. Jina embeddings 2: 8192-token general-purpose text embeddings for long documents.arXiv preprint arXiv:2310.19923 (2023)

    arXiv 2023

  6. [6]

    W., Ufkes, A., and Derpanis, K

    Harley, A. W., Ufkes, A., and Derpanis, K. G. Evaluation of deep convo- lutional nets for document image classification and retrieval. InProc. Int. Conf. on Document Analysis and Recognition (ICDAR) (2015), pp. 991–995

    work page 2015

  7. [7]

    J., Shen, Y., W allis, P., Allen-Zhu, Z., Li, Y., W ang, S., W ang, L., and Chen, W

    Hu, E. J., Shen, Y., W allis, P., Allen-Zhu, Z., Li, Y., W ang, S., W ang, L., and Chen, W. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685 (2021)

    arXiv 2021

  8. [9]

    Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825 (2023)

    arXiv 2023

Show all 21 references
  1. [10]

    Q., Sablayrolles, A., Roux, A., Mensch, A., Sa v ary, B., Bam- ford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Sa v ary, B., Bam- ford, C., Chaplot, D. S., de las Casas, D., Hanna, E. B., Bressand, F., Lengyel, G., Bour, G., Lample, G., La v aud, L. R., Saulnier, L., Lachaux, M.-A., Stock, P., Subramanian, S., Yang, S., Antoniak, S....

    2024 arXiv

  2. [11]

    Ocr-free document understanding transformer

    Kim, G., Hong, T., Yim, M., Nam, J., Park, J., Yim, J., Hw ang, W., Yun, S., Han, D., and Park, S. Ocr-free document understanding transformer. In Proc. European Conference on Computer Vision (ECCV) (2022), Springer, pp. 498–517

    2022

  3. [12]

    Building a test collection for complex document information process- ing

    Lewis, D., Agam, G., Argamon, S., Frieder, O., Grossman, D., and Heard, J. Building a test collection for complex document information process- ing. In Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval (2006)...

    2006

  4. [13]

    Roberta: A robustly optimized bert pretraining approach

    Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019)

    2019 arXiv

  5. [14]

    M., Tworek, J., Yuan, Q., Tezak, N., Kim, J

    Neelakantan, A., Xu, T., Puri, R., Radford, A., Han, J. M., Tworek, J., Yuan, Q., Tezak, N., Kim, J. W., Hallacy, C., et al. Text and code embeddings by contrastive pre-training.arXiv preprint arXiv:2201.10005 (2022)

    2022 arXiv

  6. [15]

    Cord: A consolidated receipt dataset for post-ocr parsing

    Park, S., Shin, S., Lee, B., Lee, J., Surh, J., Seo, M., and Lee, H. Cord: A consolidated receipt dataset for post-ocr parsing. InDocument Intelligence Work- shop at Neural Information Processing Systems (2019)

    2019

  7. [16]

    Im- proving language understanding by generative pre-training.OpenAI (2018)

    Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al. Im- proving language understanding by generative pre-training.OpenAI (2018)

    2018

  8. [17]

    Language models are unsupervised multitask learners

    Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog 1 , 8 (2019), 9

    2019

  9. [18]

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research 21 , 140 (2020), 1–67

    2020

  10. [19]

    Analysis of convolutional neural networks for document image classification

    Tensmeyer, C., and Martinez, T. Analysis of convolutional neural networks for document image classification. InProc. Int. Conf. on Document Analysis and Recognition (ICDAR) (2017), pp. 388–393

    2017

  11. [20]

    Huggingface’stransform- ers: State-of-the-art natural language processing.arXiv preprint arXiv:1910.03771 (2019)

    Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cis- tac, P., Rault, T., Louf, R., Funtowicz, M., et al. Huggingface’stransform- ers: State-of-the-art natural language processing.arXiv preprint arXiv:1910.03771 (2019)

    2019 arXiv

  12. [21]

    LayoutLMv2: Multi- modal pre-training for visually-rich document understanding

    Xu, Y., Xu, Y., L v, T., Cui, L., Wei, F., W ang, G., Lu, Y., Florencio, D., Zhang, C., Che, W., Zhang, M., and Zhou, L. LayoutLMv2: Multi- modal pre-training for visually-rich document understanding. InProc. Int. Joint Conference on Natural Language Processing (IJCNLP) (2021)...

    2021

  13. [22]

    MM- LLMs: recent advances in multimodal large language models

    Zhang, D., Yu, Y., Dong, J., Li, C., Su, D., Chu, C., and Yu, D. MM- LLMs: recent advances in multimodal large language models. arXiv preprint arXiv:2401.13601 (2024)

    2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.