{"id":"c821cf3a-fd1c-484e-aa3d-c9216432de57","arxiv_id":"2412.13859","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"With zero labeled examples, GPT-4-Vision reaches 69.9% accuracy on RVL-CDIP document classification, and generative LoRA fine-tuning of Mistral-7B reaches 72.5% with just 10 labels per class.","lead":"This paper tests how well large language models can classify scanned documents when given very few or no labeled examples, using a 16-class benchmark dataset. It finds that fine-tuning a 7-billion-parameter model with only 10 examples per class reaches about 72% accuracy, showing that document classification does not necessarily need hundreds of thousands of labels.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline comparison to BERT on RVL-CDIP-40K is confounded by OCR quality; the same Mistral checkpoint scores only 66.6% on the original 40K test set.","rationale":"The reader's verdict is CONDITIONAL, and my stress-test supports that conditionality without changing it. The reader identified subset representativeness as the weakest assumption, citing the Donut control as the main support and noting the OCR-quality attribution for the 40K drop. I agree that the cross-set comparison is the fragile part of the central claim, but I would sharpen it: the Donut control adequately addresses pure subset bias for image-based models, and the reader's stated concern about subset bias is partially mitigated by the (slightly higher) Donut scores on 40K. The more decisive confound is the simultaneous change in OCR quality. The paper's own 40K results with original OCR (66.6% for the classifier at 1600 samples) show a large gap from the 160x5 result (83.4%), and the paper's attribution of this gap to OCR quality is plausible but untested. A matched-OCR experiment on the same test images would isolate the cause. This is an addressable experimental gap, not a fundamental flaw, so the CONDITIONAL verdict remains appropriate. I do not see grounds for rejection: the paper is transparent about the 40K numbers, reports standard deviations for the five test sets, and does not claim state-of-the-art superiority, only that results are promising and leave room for improvement. The requested test would strengthen the paper by converting an untested attribution into a measured fact, and would clarify whether the headline comparison to BERT is fair. Thus I keep the verdict UNCHANGED but emphasize that the OCR-quality confound should be explicitly tested or the comparison should be reworded to avoid implying benchmark equivalence.","tokens_in":10455,"tokens_out":5536,"duration_ms":48103,"concrete_test":"Evaluate the 1600-sample Mistral-7B-Class checkpoint on the 800 images forming the five RVL-CDIP-160x5 test sets, but replace the Textract OCR input with the original IIT-CDIP OCR texts for those same images. Compare the resulting accuracy to the 83.4% obtained with Textract and to the 66.6% reported on the full 40K set with original OCR. If accuracy with original OCR on these 800 images drops to roughly 66-70%, the 40K drop is caused by OCR quality and the 83.4% figure is an ideal-OCR upper bound, not a direct comparison to BERT's 85.0%. If accuracy remains near 80%, then the 160x5 subsets are easier for text-based classifiers, and the Donut-based representativeness argument fails for OCR-based models.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central comparative claim juxtaposes 83.4% few-shot accuracy on the custom RVL-CDIP-160x5 test sets with an 85.0% fully trained BERT result on the full RVL-CDIP-40K test set. This is not an apples-to-apples comparison because two factors change simultaneously: the test subset and the OCR source. The 160x5 sets use high-quality Amazon Textract OCR, while the 40K result uses the original IIT-CDIP OCR, which the paper itself describes as 90s-era and 'unquestionably' lower quality. The paper reports its own Mistral-7B-Class model on the original 40K set with original OCR: only 66.6% at 1600 samples. The 16.8-point gap between 83.4% and 66.6% is attributed to OCR quality, but this attribution is not tested directly. The Donut control does show the 160x5 subsets are not biased for image-based models (Donut scores 76.4% on 40K vs 73.8% on 160x5), so subset representativeness is partially supported. However, Donut is OCR-free and cannot control for the OCR-quality confound that affects all text-based models. Without a matched-OCR evaluation, the equivalence between the 160x5 and 40K benchmarks is unestablished for OCR-based methods, and the headline '83.4% vs 85.0%' may overstate how the few-shot model would perform on the standard RVL-CDIP benchmark under realistic OCR conditions. The internal rankings on the 160x5 sets remain valid, but the cross-set comparison to BERT is the load-bearing weakness.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper benchmarks zero-shot prompting and few-shot fine-tuning for document image classification on RVL-CDIP using a range of models: generative LLMs (Mistral-7B, Mixtral, GPT-3.5, GPT-4, GPT-4-Vision), text embeddings with KNN, fine-tuned RoBERTa, LoRA fine-tuned Mistral-7B in classifier and generative modes, and the image-based Donut model. The authors introduce RVL-CDIP-160x5, five random 160-sample test subsets with Amazon Textract OCR, and report mean accuracy and standard deviation over the subsets. Their headline findings are that generative LoRA fine-tuning of Mistral-7B reaches 72.5% accuracy with ten samples per class, classifier fine-tuning reaches 83.4% with 100 samples per class, and they compare this with an 85.0% fully trained BERT result on RVL-CDIP-40K from the literature. They conclude that zero-shot and few-shot LLM-based document classification is feasible and that generative fine-tuning is sample-efficient.","tokens_in":10809,"tokens_out":5390,"duration_ms":49296,"significance":"If the claims hold, the paper provides a useful benchmark for few-shot document classification and a careful set of comparisons across prompting, embedding, and fine-tuning approaches. The experimental design has notable strengths: five random test subsets with mean and standard deviation, validation-based selection of prompts and hyperparameters, and inclusion of both the custom 160x5 test sets and the full 40K test set as a reference. The headline comparison with BERT is striking and would be of broad interest, but as reported it is confounded by simultaneous changes in test subset and OCR quality. The internal comparisons on RVL-CDIP-160x5 appear sound; the cross-benchmark contextual claim needs additional support.","major_comments":[{"comment":"The central contextual claim that 83.4% few-shot Mistral-7B-Class accuracy is 'a notable achievement when compared to the 85.0% accuracy reported in [2]' is confounded by two factors that change simultaneously: the test set and the OCR engine. RVL-CDIP-160x5 uses Amazon Textract OCR, while RVL-CDIP-40K uses the original IIT-CDIP OCR, which the paper itself describes in Section 4.1 as 'unquestionably' lower quality. The same Mistral-7B-Class checkpoint scores only 66.6% on RVL-CDIP-40K (Table 5), and the paper attributes the 16.8-point gap to OCR quality, but this attribution is not tested directly. A matched-OCR evaluation, for example running Textract on a random subset of the 40K test set or running the original OCR on the 160x5 documents, is needed before the '83.4% vs 85.0%' comparison can support the stated conclusion. As written, the comparison is not established.","section":"Section 4.4, Table 6"},{"comment":"The conclusion that 'the five-fold selection of 160 test samples is, indeed, representative' is supported only by the OCR-free Donut control. Donut's similar accuracy on RVL-CDIP-40K and RVL-CDIP-160x5 controls for image-content bias, but it does not control for the OCR-quality shift that affects all OCR-based models in Table 5. The systematic drop observed for RoBERTa and Mistral-7B on RVL-CDIP-40K may indeed be due to inferior OCR, but it could also be caused by a subset bias that interacts with text-based methods. The representativeness claim should either be restricted to image-based models or be backed by an OCR-aware check on the same documents under both OCR conditions.","section":"Section 4.4"},{"comment":"The RVL-CDIP-40K results are reported as single-run values without standard deviations or repetition, even though they are used to support the OCR-quality hypothesis and serve as the reference for the headline comparison with BERT. Since the 160x5 results all carry mean and standard deviation, the asymmetry in reporting makes it difficult to assess the reliability of the 40K numbers. Please report uncertainty across repeated runs, or at minimum explicitly label these values as exploratory single-run references and avoid drawing strong conclusions from small differences on this set.","section":"Section 4.1, Table 5"}],"minor_comments":[{"comment":"There is a typo in 'specifying a temparature of0': 'temperature' is misspelled and a space is missing before the zero.","section":"Section 4.1"},{"comment":"The sentence introducing Table 6 says the 160x5 results are 'put into context' with literature results on RVL-CDIP-40K; it should explicitly state that this context comparison crosses both test-set definitions and OCR sources, since that is exactly the caveat raised in the major comments.","section":"Section 4.5, Table 6"},{"comment":"References [8] and [9] both point to the same Mistral 7B paper (arXiv:2310.06825); one duplicate should be removed and the in-text citations merged.","section":"References"},{"comment":"The t-SNE plot would be more reproducible if the parameters (perplexity, number of iterations, initialization) were reported, or if the figure is intended as purely illustrative this should be stated.","section":"Figure 3"},{"comment":"Several models produce high invalid-answer rates; a sentence explicitly stating that invalid answers are counted as incorrect (rather than excluded from the accuracy computation) would help readers interpret the reported accuracy numbers.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a useful empirical benchmark, and the measured results on RVL-CDIP-160x5 appear carefully obtained with appropriate validation and uncertainty reporting. My main concern is the confounded cross-benchmark comparison to the 85.0% BERT result: the test-set and OCR variables change together, and the Donut control cannot resolve this for text-based models. This is fixable by adding a matched-OCR experiment or by substantially softening the comparison claim, so I do not recommend rejection. I would encourage the editor to require the authors to address Major Comment 1 directly rather than only reword the conclusion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis is a benchmark paper, and it says so up front: no new method. What it does offer are useful measurements. The cleanest new result is that generative LoRA fine-tuning of Mistral-7B reaches 72.5% mean accuracy on RVL-CDIP with ten samples per class, and that a classifier head overtakes it at 100 samples per class, reaching 83.4%. That direct comparison between generative and classifier fine-tuning on the same base model is genuinely informative. The RVL-CDIP-160x5 protocol—five random 160-sample test sets, mean and standard deviation, validation-based hyperparameter selection—is a sensible way to keep API costs down while still getting some error bars. The paper is also transparent about OCR quality differences. I largely agree with the reader's conditional verdict.\n\nThe soft spot is the cross-set comparison to fully trained BERT. The headline '83.4% vs 85.0%' compares results on the custom 160x5 sets with modern Textract OCR to a BERT figure on the full RVL-CDIP-40K set with the original 90s-era IIT-CDIP OCR. Two variables change at once: test subset and OCR source. The stress-test note gets this right, and the paper's own 40K results show the same Mistral-7B classifier dropping to 66.6%. That drop is plausibly due to OCR quality, but it is not directly tested. Donut supports subset representativeness for image-based models, not for OCR-based text models. So the internal rankings on 160x5 stand, but the claim that few-shot text classification nearly matches a fully trained model on the standard benchmark overstates what was measured.\n\nMinor issues: the 40K numbers are single-run and come without error bars, and the code and benchmark data are promised but not yet shipped. The prompt selection on a validation set is mild but not circular. The citation pattern looks fine.\n\nI would send this to peer review. It is a useful reference point for the document classification community, and the weaknesses are addressable in revision: release the code and data, run a matched-OCR comparison on a subset of the 40K set, and report error bars there. A serious referee should treat the OCR-quality explanation as a hypothesis to be tested, not a settled conclusion.","headline":"A genuinely useful few-shot document classification benchmark whose headline comparison to fully trained BERT on RVL-CDIP-40K is not apples-to-apples because the test sets and OCR quality change together.","tokens_in":11326,"tokens_out":3210,"would_cite":true,"duration_ms":28766,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper establishes that large language models can classify scanned documents with almost no training data: generative fine-tuning of Mistral-7B on ten samples per class reaches 72.5% accuracy on RVL-CDIP, and 100 samples per class…","keywords":["Document Image Classification","OCR","Large Language Models","Few-Shot Learning","Zero-Shot Prompting","Fine-Tuning","LoRA","RVL-CDIP"],"falsifier":"Take the 1,600-sample fine-tuned classifier and generative model, run them on the full RVL-CDIP-40K test set using the same high-quality OCR used for the small test sets, and compare accuracy on the 40K set to the 83.4% and 72.5% figures; a large drop would falsify the representativeness assumption. Alternatively, compute accuracy separately for documents with low OCR confidence to test whether the small subsets skew toward legible documents.","tokens_in":10277,"feed_emoji":"📄","tokens_out":5149,"duration_ms":39719,"temperature":0.7,"pith_summary":"This paper asks whether document image classification can be done with almost no labeled training data, using large language models that read OCR text. It reports a benchmark on RVL-CDIP showing that a zero-shot multimodal model reaches 69.9% accuracy, and that generative fine-tuning of Mistral-7B with only ten samples per class reaches 72.5% accuracy on a five-fold 160-sample test selection. With 100 samples per class, classifier fine-tuning reaches 83.4%, close to the 85.0% reported for a fully trained BERT model using 320,000 samples. The paper's point is that LLMs collapse the data requirement for document classification by orders of magnitude, while leaving room for improvement.","feed_headline":"Ten examples per class teach an LLM document classification at 72.5%","feed_subtitle":"Zero-shot and few-shot LLM results rival fully trained models, slashing annotation needs.","key_machinery":"The argument rests on a purpose-built evaluation scenario, RVL-CDIP-160x5, consisting of five disjoint 160-sample subsets (ten per class) drawn from the RVL-CDIP test set, with fresh high-quality OCR from Amazon Textract. The fine-tuning results are produced by quantized LoRA adaptation of Mistral-7B (4-bit base weights, rank-8 adapters) in two modes: generative, where the model predicts a class token appended after a 'Classification:' directive, and classifier, where a classification head replaces the language-model head. Zero-shot prompting uses a system prompt refined with GPT-4, evaluated on OCR text or document images (GPT-4-Vision). The RVL-CDIP-40K test set with older OCR is retained as a reference to check how representative the small subsets are.","core_discovery":"The central empirical discovery is that generative fine-tuning of a small open-source LLM, Mistral-7B, with a LoRA adapter on OCR text is extraordinarily sample-efficient for document classification. Using only 160 training documents (ten per class), it reaches 72.5% mean accuracy on the RVL-CDIP-160x5 test sets, outperforming fully fine-tuned RoBERTa (59.8%), KNN embeddings, and image-only Donut. With 1,600 samples (100 per class), a classifier head on the same base model reaches 83.4%, approaching the 85.0% accuracy of a fully trained BERT model on the full RVL-CDIP-40K test set. In zero-shot prompting, GPT-4-Vision, which sees only document images, reaches 69.9% without any training samples. The comparison establishes that both zero-shot prompting and few-shot fine-tuning are viable regimes for document classification.","pith_inferences":["The representativeness of the RVL-CDIP-160x5 subsets is the main risk: if the five 160-sample draws are easier than the full test set, the 72.5% and 83.4% figures overstate real-world accuracy. A direct evaluation on the full 40K test set with matched high-quality OCR would settle this.","Because the generative model was trained only on OCR text, it ignores layout and image cues; combining the LoRA-tuned text model with a visual encoder might push few-shot accuracy closer to the 97% reported for fully trained multimodal systems.","The benchmark design could be extended to out-of-distribution document types to test whether the few-shot gains transfer to categories absent from pretraining, rather than only the 16 RVL-CDIP classes."],"forward_implications":["If the results hold, document classification pipelines can be built with tens to hundreds of labeled documents instead of hundreds of thousands.","Zero-shot prompting with a multimodal LLM can serve as an immediate baseline for new document categories before any annotation effort.","Generative fine-tuning is preferable to classifier fine-tuning when training data are very scarce, because it retains output flexibility and is more stable across sample sizes.","The 83.4% result with 100 samples per class implies that LLM-based classifiers can rival fully trained specialized models on RVL-CDIP, despite being an order of magnitude larger in parameters.","OCR quality is a key variable: the gap between RVL-CDIP-160x5 and RVL-CDIP-40K results for OCR-based models shows that high-quality OCR is needed to realize these accuracies."],"supporting_citations":[{"why":"Supplies the RVL-CDIP dataset and its train/validation/test split, which is the benchmark everything is evaluated on.","marker":"[6]"},{"why":"Introduces LoRA, the parameter-efficient fine-tuning method used for Mistral-7B in the few-shot experiments.","marker":"[7]"},{"why":"Provides Mistral-7B, the base LLM for both generative and classifier fine-tuning.","marker":"[8]"},{"why":"Provides Donut, the OCR-free image model used as a comparison and as the main evidence for the representativeness of the RVL-CDIP-160x5 subsets.","marker":"[11]"},{"why":"Reports the fully trained BERT 85.0% and BERT+NasNet 97.1% accuracies that contextualize the few-shot results.","marker":"[2]"},{"why":"Describes QLoRA, the 4-bit quantization method that makes LoRA fine-tuning of Mistral-7B feasible on standard hardware.","marker":"[4]"},{"why":"Documents GPT-4 and GPT-4-Vision, which are the models used for zero-shot prompting with text and images.","marker":"[1]"},{"why":"Provides the IIT-CDIP collection and the original low-quality OCR texts used for the RVL-CDIP-40K reference test set.","marker":"[12]"}],"fun_headline_variants":["160 docs: LLM hits 72.5% on document classification","Zero-shot GPT-4V sees documents at 69.9% accuracy","Few-shot LLM rivals full models with 83.4% on RVL-CDIP","LLM document classification: 10 samples per class enough","Sixty-nine zero-shot, 72.5% few-shot: LLM document smarts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline accuracy numbers rest on the assumption that the five randomly chosen 160-document test subsets fairly represent the full RVL-CDIP test set, so that results on them transfer to real-world document streams.","fun_headline_variants_meta":{"raw":{"variants":["160 docs: LLM hits 72.5% on document classification","Zero-shot GPT-4V sees documents at 69.9% accuracy","Few-shot LLM rivals full models with 83.4% on RVL-CDIP","LLM document classification: 10 samples per class enough","Sixty-nine zero-shot, 72.5% few-shot: LLM document smarts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000467,"raw_usage":{"total_tokens":2285,"prompt_tokens":858,"completion_tokens":1427,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":474,"completion_tokens_details":{"reasoning_tokens":1324}},"tokens_in":474,"tokens_out":1427,"duration_ms":10838,"temperature":1.0,"reasoning_tokens":1324,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:42:53.920091+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the 1,600-sample fine-tuned classifier and generative model, run them on the full RVL-CDIP-40K test set using the same high-quality OCR used for the small test sets, and compare accuracy on the 40K set to the 83.4% and 72.5% figures; a large drop would falsify the representativeness assumption. Alternatively, compute accuracy separately for documents with low OCR confidence to test whether the small subsets skew toward legible documents.","supporting_citations":[{"cited_title":"W., Ufkes, A., and Derpanis, K","cited_arxiv_id":null,"evidence_quote":"Supplies the RVL-CDIP dataset and its train/validation/test split, which is the benchmark everything is evaluated on."},{"cited_title":"Ocr-free document understanding transformer","cited_arxiv_id":null,"evidence_quote":"Provides Donut, the OCR-free image model used as a comparison and as the main evidence for the representativeness of the RVL-CDIP-160x5 subsets."},{"cited_title":"Visual and textual deep feature fusion for document image classification","cited_arxiv_id":null,"evidence_quote":"Reports the fully trained BERT 85.0% and BERT+NasNet 97.1% accuracies that contextualize the few-shot results."},{"cited_title":"Qlora: Efficient finetuning of quantized llms.Advances in Neural Information Processing Systems 36 (2024)","cited_arxiv_id":null,"evidence_quote":"Describes QLoRA, the 4-bit quantization method that makes LoRA fine-tuning of Mistral-7B feasible on standard hardware."},{"cited_title":"Building a test collection for complex document information process- ing","cited_arxiv_id":null,"evidence_quote":"Provides the IIT-CDIP collection and the original low-quality OCR texts used for the RVL-CDIP-40K reference test set."}],"review_version":1}