{"id":"76313e64-ef98-4a67-a53c-670a248a0317","arxiv_id":"2411.14957","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A 7B LLaVA model fine-tuned on Claude 3 Sonnet's synthetic labels matches Sonnet's extraction accuracy on sharp expense receipts while cutting cost by 85% and increasing speed 5x.","lead":"The paper trains a small open-source vision-language model on labels generated by Claude 3 Sonnet, then shows the student can match the teacher on high-quality expense receipts at roughly 85% lower cost and 5x faster. The method, called TAIL, targets companies extracting merchant, amount, and date fields from millions of receipts that lack labeled training data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"At-par claim lacks exact-label validation: CORD shows LLaVA-Net well below Sonnet, and the claimed image-quality condition is never tested with Sonnet on the stratified subsets.","rationale":"The reader's weakest assumption correctly identifies the noisy-label problem, but the more specific and load-bearing gap is that the paper's own exact-label benchmark (CORD) contradicts the at-par claim, and the image-quality condition that is supposed to rescue it is never directly tested against Sonnet on exact labels. The 'at par or better' claim therefore rests entirely on an internal dataset with unverifiable labels, while the one externally validated comparison shows the student underperforming the teacher by 10–24 points. This does not invalidate the distillation methodology, which shows real gains over layout-aware baselines and over training on noisy labels, but it means the headline claim is conditional on a validation that has not been performed. The proposed test would settle whether the condition actually holds. The verdict remains CONDITIONAL rather than REJECT because the method has independent value and the missing analysis is well-defined and feasible; the paper should not be accepted as-is without that stratified exact-label comparison.","tokens_in":14128,"tokens_out":5123,"duration_ms":49235,"concrete_test":"Using the CORD test set (exact labels), stratify by Laplacian variance of the receipt image. For the subset with variance ≥ 200, compute ANLS (M=1) and tree-edit distance (M=24) for both Claude 3 Sonnet zero-shot with TAIL prompts and LLaVA-Net. If LLaVA-Net's scores are within a small margin (e.g., ≤2 ANLS points) of Sonnet's, the claimed condition is validated; if it remains substantially below (as the overall 84 vs 94 suggests), the at-par claim fails on exact labels, and the ExpenseQA result is an artifact of noisy-label evaluation. Also report the same stratified comparison for the internal ExpenseQA if possible, using a human-verified subsample.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 5.3.3 (RQ3) asserts that on images with Laplacian variance ≥ 200, LLaVA-Net achieves ANLS exceeding 90%, 'almost at par with Sonnet,' but Table 5 reports only LLaVA-Net and LayoutLMV3; Sonnet's scores on the same quality-stratified subsets are never given. On the only benchmark with exact labels (CORD), Table 3 shows LLaVA-Net at 84% vs Sonnet's 94% (M=1) and 69% vs 93% (M=24), far from at par, and no quality-stratified exact-label comparison is provided. The abstract's claim that the external dataset demonstrates conditions of at-par performance is therefore unsupported. The at-par claim for ExpenseQA is measured only against employee-typed noisy labels, which the paper itself shows systematically disagree with the receipt image (Sec. 3, Fig. 1). Because TAIL labels track the image rather than the typed fields, both Sonnet and the student can score high ANLS against noisy labels only when the noisy label happens to match; the 52%/70%/83% vs 50%/70%/76% differences could reflect different disagreement patterns rather than true extraction accuracy. Without human-verified labels on ExpenseQA, and without a Sonnet-vs-LLaVA-Net comparison on the high-quality CORD subset, the 'at par or better' claim is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TAIL (Task Aware Instruction-based Labelling), which uses Claude 3 Sonnet to generate synthetic labels for visually rich documents without ground truth, and distills these labels into a 7B LLaVA-based student (LLaVA-Net) via response-based knowledge distillation without using the teacher's weights or training data. The authors evaluate LLaVA-Net on an internal ExpenseQA dataset (merchant name, amount, transaction date) against employee-typed noisy labels, and on the CORD benchmark with exact labels under M=1 and M=24 settings, reporting ANLS and tree-edit distance scores. They claim the student matches or exceeds Sonnet on ExpenseQA while being about 85% cheaper and roughly 5x faster, and they demonstrate an overpayment-detection use case.","tokens_in":14402,"tokens_out":4798,"duration_ms":41717,"significance":"If the central claims hold, the work would be practically valuable: it shows a feasible pipeline for training small open models on proprietary document collections with only API access to a teacher, without sharing sensitive data, and it quantifies cost and latency benefits. Strengths include the use of an external benchmark with exact labels (CORD), the explicit acknowledgment that employee-typed fields are noisy (Sec. 3, Fig. 1), the RQ4 analysis showing the student is not a literal replica of Sonnet (87/74/82% ANLS with Sonnet, zero refusals vs 3%), and the cost/speed comparison. However, the main 'at par or better' claim depends on a validation set whose ground truth is itself shown to be unreliable, and the exact-label evidence on CORD points in the opposite direction.","major_comments":[{"comment":"The central claim that LLaVA-Net performs 'at par or better' than Sonnet on ExpenseQA is measured against employee-typed noisy labels, which the paper itself shows are frequently inconsistent with the receipt image (Sec. 3, Fig. 1, Tab. 2). Because TAIL labels are designed to match the image rather than the typed fields, ANLS against noisy labels does not measure true extraction accuracy; a high score can be achieved only when the noisy label happens to match the image. The reported differences (52% vs 50% merchant, 70% vs 70% amount, 83% vs 76% date) may reflect different disagreement patterns with noisy labels rather than better extraction. No human-verified ground truth is provided for ExpenseQA, so the abstract's at-par claim is not established.","section":"Sec. 5.3.1, Table 3"},{"comment":"The paper asserts that on images with Laplacian variance >= 200, LLaVA-Net's ANLS exceeds 90%, 'almost at par with Sonnet,' but Table 5 reports only LLaVA-Net and LayoutLMV3; Sonnet's scores on the same quality-stratified subsets are never given. On the only benchmark with exact labels (CORD), Table 3 shows LLaVA-Net at 84% vs Sonnet's 94% (M=1) and 69% vs 93% (M=24), far from at par. The abstract's claim that the external dataset demonstrates conditions of at-par performance is therefore unsupported. A quality-stratified exact-label comparison with Sonnet would be needed to substantiate the stated condition.","section":"Sec. 5.3.3, Table 5"},{"comment":"All results are reported as point estimates without confidence intervals, significance tests, or multiple seeds. Given the close margins on ExpenseQA (e.g., 52% vs 50% for merchant name) and the small validation set (n=1000), the differences may be within noise. The paper should report variance estimates or at least state the number of runs; this is especially important because the headline 'at par or better' claim depends on these comparisons.","section":"Tables 3-5"}],"minor_comments":[{"comment":"The notation M=1 for ExpenseQA is confusing, since three attributes (merchant name, amount, date) are extracted; please clarify whether M denotes the number of attributes per query or the number of output fields.","section":"Sec. 3"},{"comment":"The dash for LayoutLMV3 on transaction date is explained in the text, but a footnote in the table would improve readability and prevent readers from interpreting it as an omission.","section":"Table 3"},{"comment":"There are several naming inconsistencies: 'LLaV A-Net', 'LlaV A-Net', 'LLaVA-Net', and 'LLaV A' are used interchangeably; please unify the notation.","section":"Throughout"},{"comment":"Minor typos: 'V AT' should be 'VAT' in the introduction, and 'multimodel student model' in the conclusion should be 'multimodal student model'.","section":"Introduction and Conclusion"},{"comment":"The overpayment detection result (89 invoices, about $2 per document) is reported without context on the total number of invoices assessed or the distribution of discrepancies; adding this context would strengthen the use-case claim.","section":"Sec. 5.3.5"},{"comment":"The exact internal prompts are withheld for confidentiality, which is understandable, but the paper should state more explicitly which parts of the pipeline are reproducible by others (e.g., CORD prompts are in the supplementary material, while ExpenseQA prompts are not).","section":"Sec. 4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is an industry-oriented empirical study. The main scientific risk is that the headline evaluation uses noisy labels as ground truth, and the only exact-label benchmark (CORD) contradicts the at-par claim. I would urge the editor to require either a human-annotated gold subset of ExpenseQA or a substantial revision of the claims. The paper could be acceptable if the claims are appropriately scoped and the missing Sonnet stratified comparison is added. No concerns about citation behavior or novelty beyond the above."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this paper describes a practical pipeline for training a 7B open model to extract fields from expense receipts using synthetic labels from Claude Sonnet, and the engineering is real. But the headline claim that the student matches Sonnet is only supported on internal data with noisy employee-typed labels, and the one exact-label benchmark tells the opposite story: LLaVA-Net gets 84% vs Sonnet's 94% on CORD M=1, and 69% vs 93% for M=24. That is not at par.\n\nWhat's genuinely new here is the combination of task-aware synthetic labeling (TAIL) with response-based distillation into LLaVA, and the empirical observations around image quality and rare formats. The paper shows that layout-aware models like LayoutLMV3 fail on heterogeneous receipts and that a distilled multimodal model can handle reasoning about context, e.g., detecting overpayment when the typed amount doesn't match the receipt. RQ4 is also honest: the student isn't a simple replica, with lower refusal rates and different error patterns.\n\nThe soft spots are real but not fatal if you read carefully. The ExpenseQA evaluation uses noisy labels that the paper itself shows disagree with the image (Fig. 1, Tab. 2). Because TAIL labels match the image, ANLS against noisy labels is a murky measure of extraction accuracy. The image-quality condition (Laplacian variance >=200) is introduced post hoc, and Table 5 only gives LLaVA-Net and LayoutLMV3; Sonnet's scores on the quality-stratified subsets are never reported, so the claim of \"almost at par\" is unsupported. Also, no confidence intervals, internal prompts and data are withheld, and code isn't released. These weaken the scientific contribution but don't negate the practical value.\n\nWho should read this? Anyone building a production system for document extraction with a budget constraint will find the cost/latency numbers and the guidance on image quality useful. Academic readers should treat the at-par claim as a hypothesis to test, not a demonstrated fact.\n\nVerdict: worth a serious referee. The pipeline is coherent, the evaluation is at least partially honest, and the rare-format and overpayment analyses are useful. The authors should be pushed to provide Sonnet scores on the stratified CORD subsets, a hand-verified sample of ExpenseQA, or a revised claim that makes the noisy-label dependence explicit.","headline":"A useful distillation pipeline whose at-par claim is only supported on noisy internal labels and contradicted by the exact-label benchmark.","tokens_in":14953,"tokens_out":2408,"would_cite":true,"duration_ms":21883,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A small model trained on synthetic labels matches a state-of-the-art commercial large multimodal model on expense-document extraction, with 85% lower cost and about 5x faster inference.","keywords":["visually rich document understanding","synthetic label generation","knowledge distillation","multimodal large language models","expense receipt extraction","task-aware prompting","noisy labels","ANLS"],"falsifier":"Take a random sample of, say, 500 internal receipts for which the employees typed merchant name, amount, and date; have human annotators transcribe the true values from the images; then compute ANLS for LLaVA-Net, Sonnet, and the noisy labels against these human-verified values. If LLaVA-Net's human-verified ANLS falls significantly below Sonnet's, the central at-par claim fails; if noisy labels score lower than both models, the paper's premise that noisy labels are a usable benchmark is also weakened.","tokens_in":13903,"feed_emoji":"🧾","tokens_out":4778,"duration_ms":35828,"temperature":0.7,"pith_summary":"The paper argues that organizations can build accurate information-extraction systems for heterogeneous receipts and invoices even when no ground-truth labels exist. It introduces TAIL, a prompting protocol that gets a large commercial multimodal model (Claude 3 Sonnet) to produce synthetic labels in a required output format, and then fine-tunes a small open-source vision-language model, LLaVA-Net, on those labels using response-based knowledge distillation. On an internal expense dataset of a multinational company, the distilled model matches or slightly beats the teacher on merchant name, amount, and date extraction while costing about 85% less and running about five times faster. The paper also shows the student outperforms layout-aware baselines on heterogeneous documents, and demonstrates how the extracted fields can flag overpayment risk.","feed_headline":"Small model beats Claude 3 on receipts at 85% lower cost","feed_subtitle":"TAIL synthetic labels let a distilled 7B model match its teacher on expense documents, ~5x faster.","key_machinery":"The load-bearing mechanism is TAIL (Task Aware Instruction-based Labelling), a set of task-specific prompt templates that constrain the teacher's output to the exact format required for each field (e.g., 'only the value, no extra words'). These teacher outputs serve as soft labels for response-based knowledge distillation, in which a LLaVA model (CLIP vision encoder plus a Mistral-based language model) is fine-tuned with LoRA. The paper also uses the Laplacian variance of receipt images as a quality gate: the at-par claim holds when most images are sharp (variance ≥ 200), and the student's advantage over layout-aware models is attributed to its reasoning ability rather than template matching.","core_discovery":"On the paper's own terms, the central discovery is that synthetic labels from a state-of-the-art LMM, generated without access to the teacher's weights or training data, can transfer enough task knowledge to a much smaller student model for it to reach parity with the teacher on in-domain receipt extraction. Specifically, LLaVA-Net achieves ANLS of 52% on merchant name, 70% on amount, and 83% on date against noisy employee-typed labels, versus Sonnet's 50%, 70%, and 76%, under the condition that most images have Laplacian variance at or above 200. The student also shows fewer refusals and fewer format violations than the teacher, and produces the desired output format exclusively.","pith_inferences":["A natural next step the paper leaves implicit is evaluating both teacher and student against human-verified ground truth on the internal corpus; without that, the at-par claim is only relative to noisy labels.","The same TAIL-and-distill recipe could transfer to other structured-document domains (contracts, medical records, bank forms) whenever a closed LMM is acceptable as a one-time labeler and a small internal model is wanted for production.","One could iterate the procedure: use LLaVA-Net itself to label fresh document streams and periodically distill back, which the paper does not explore.","A testable extension is to measure whether the student's higher ANLS on amount and date reflects genuinely better reading or simply overfitting to the noisy-label distribution."],"forward_implications":["Expense teams can train custom extraction models for fields that employees do not type in, since TAIL supplies labels where no noisy labels exist.","Deploying the 7B-parameter student on a single GPU instance instead of calling a closed API reduces annual cost by roughly 85% and raises throughput about fivefold, making large-scale receipt auditing more feasible.","Layout-aware models such as LayoutLMv3 trained on the same synthetic labels lag the student by more than 10% ANLS on heterogeneous corpora, because they fail on rare receipt templates and rely on fragile OCR.","Because the student rarely refuses and rarely deviates from the requested format, downstream automation (policy checks, duplicate detection, overpayment flags) can trust its output structure.","An overpayment-detection case study shows that comparing model-extracted amounts to employee-typed amounts surfaces invoices where reimbursement likely exceeds the receipt, at about $2 per flagged document."],"supporting_citations":[{"why":"Supplies the only external ground-truth benchmark (CORD) for validating the approach against exact labels.","marker":"[44]"},{"why":"Provides the LLaVA architecture that is fine-tuned as the student model.","marker":"[31]"},{"why":"Defines response-based knowledge distillation, the training mechanism used to transfer teacher knowledge.","marker":"[20]"},{"why":"Identifies Claude as the teacher model whose outputs become TAIL labels.","marker":"[4]"},{"why":"LayoutLMv3 is the layout-aware baseline the paper compares against and outperforms.","marker":"[23]"},{"why":"Defines ANLS, the evaluation metric used for all extraction comparisons.","marker":"[9]"},{"why":"Describes LoRA, the efficient fine-tuning method used to train the student.","marker":"[21]"},{"why":"Cites Amazon Bedrock's data-protection escrow to justify using a closed teacher on sensitive internal receipts.","marker":"[2]"}],"fun_headline_variants":["Distilled 7B model beats Claude 3 on receipts at 85% lower cost","Synthetic labels let small model match Claude 3 on expense docs","TAIL: No-label training beats Claude 3 at fraction of cost","7B model tops Claude 3 on receipts, 5x faster","Zero-label distillation matches Claude 3 on receipts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The at-par comparison on internal data relies on employee-typed noisy labels as the yardstick, even though the paper shows these labels often disagree with the receipt images; if those noisy labels are biased toward one model, the claimed parity could be an artifact.","fun_headline_variants_meta":{"raw":{"variants":["Distilled 7B model beats Claude 3 on receipts at 85% lower cost","Synthetic labels let small model match Claude 3 on expense docs","TAIL: No-label training beats Claude 3 at fraction of cost","7B model tops Claude 3 on receipts, 5x faster","Zero-label distillation matches Claude 3 on receipts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000213,"raw_usage":{"total_tokens":1449,"prompt_tokens":1002,"completion_tokens":447,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":618,"completion_tokens_details":{"reasoning_tokens":353}},"tokens_in":618,"tokens_out":447,"duration_ms":4098,"temperature":1.0,"reasoning_tokens":353,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:40:17.533420+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of, say, 500 internal receipts for which the employees typed merchant name, amount, and date; have human annotators transcribe the true values from the images; then compute ANLS for LLaVA-Net, Sonnet, and the noisy labels against these human-verified values. If LLaVA-Net's human-verified ANLS falls significantly below Sonnet's, the central at-par claim fails; if noisy labels score lower than both models, the paper's premise that noisy labels are a usable benchmark is also weakened.","supporting_citations":[{"cited_title":"Cord: A con- solidated receipt dataset for post-ocr parsing","cited_arxiv_id":null,"evidence_quote":"Supplies the only external ground-truth benchmark (CORD) for validating the approach against exact labels."},{"cited_title":"Introducing the next generation of claude, 2024","cited_arxiv_id":null,"evidence_quote":"Identifies Claude as the teacher model whose outputs become TAIL labels."},{"cited_title":"Layoutlmv3: Pre-training for document ai with unified text and image masking, 2022","cited_arxiv_id":null,"evidence_quote":"LayoutLMv3 is the layout-aware baseline the paper compares against and outperforms."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines ANLS, the evaluation metric used for all extraction comparisons."},{"cited_title":"Data protection, 2024","cited_arxiv_id":null,"evidence_quote":"Cites Amazon Bedrock's data-protection escrow to justify using a closed teacher on sensitive internal receipts."}],"review_version":1}