{"id":"ce4658e8-837e-4a56-ba1c-461dfe07b818","arxiv_id":"2608.08727","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"TomaMMU is a tomato-leaf-disease VQA dataset with more than 200,000 question-answer pairs, and TomaBench shows that current vision-language models underperform on it, while fine-tuning on TomaMMU lifts MCQ accuracy to about 96%.","lead":"The authors introduce TomaMMU, a large question-answer dataset for tomato leaf disease images, and TomaBench, a seven-task benchmark for vision-language models. Most tested models score far below expert level on the benchmark, while fine-tuning a model on TomaMMU raises its MCQ accuracy to about 96%.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No disjointness check between TomaBench and the 124,367 fine-tuning QA pairs; the 96.09% fine-tuning result is uninterpretable until leakage is ruled out, and the paper's own TomaBench size numbers conflict.","rationale":"The paper makes a useful dataset contribution, and much of its zero-shot evaluation is plausible; the conditional verdict from the reader is appropriate. The single most load-bearing assumption, however, is split integrity. The 96.09% MCQ result advertised in the abstract is only meaningful if TomaBench is out-of-sample relative to the 124,367 pairs used to fine-tune TomaLLaVA. The manuscript asserts this informally but provides no verification, and its own counts conflict: Section 3.1 implies an evaluation set of 80,442 QA pairs, while the Introduction and Conclusion say 42,626. Because the questions are generated from per-image metadata, exact or near-duplicate image leakage is a concrete failure mode, and the magnitude of the fine-tuning gain amplifies the concern. I am not alleging intentional leakage; the issue is that the evidence needed to rule it out is absent and easy to produce. The concrete test I propose—image-level and text-level overlap analysis on the released data, followed by retraining if overlap is found—would settle the question directly. If the check passes, the central claim stands; if it fails, the headline result is invalid. For these reasons the reader's CONDITIONAL verdict remains unchanged.","tokens_in":16539,"tokens_out":5966,"duration_ms":60702,"concrete_test":"Download the released TomaMMU/TomaBench files from HuggingFace. Reconstruct the actual split by reconciling the 42,626 vs. 42,626+37,816 count discrepancy. Then: (1) compute exact and near-duplicate image overlap between the fine-tuning set and TomaBench using perceptual hashes (e.g., pHash) and a feature embedding (e.g., DINOv2) with a similarity threshold; (2) compute normalized question-stem and option overlap (e.g., MinHash on tokenized MCQs), and check whether the same image contributes QA pairs to both splits. If any overlap exists, re-run the TomaLLaVA fine-tuning protocol on a strictly disjoint subset and report the resulting MCQ accuracy; if the numbers still do not reconcile, state explicitly which QA pairs constitute TomaBench.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that TomaBench exposes pronounced gaps and that simple fine-tuning on TomaMMU closes the gap to 96.09% on MCQs—requires TomaBench to be a genuinely held-out evaluation set. The manuscript never verifies this. Section 3.1 states a training set of 124,367 VQA pairs and 'an evaluation set with 42,626 MCQs and 37,816 OEQs ... called TomaBench'; the Introduction and Conclusion instead say TomaBench contains 42,626 question-answer pairs. These figures are mutually inconsistent, so the exact size and composition of the benchmark are not established. Section 3.2 Stage 3 says only that 'a small, balanced subset, TomaBench, is selected for evaluation, while the remaining data is used for fine-tuning and training'—an assertion of intent, not a demonstrated property. No image-level deduplication, no near-duplicate analysis, and no question/answer overlap analysis is reported. Because TomaBench questions are templated from per-image metadata (one MCQ per symptom, merged OEQ per image), a single shared or near-duplicate image between training and test would allow the fine-tuned model to memorize the associated metadata rather than exhibit diagnostic understanding. The reported gain (96.1% vs. 61.3% for the best zero-shot model and 70.3% for Gemini 2.0 Flash) is the pattern leakage would produce. Until disjointness is established, the headline result cannot support the paper's central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces TomaMMU, a tomato-leaf disease multimodal understanding dataset with 28,808 images across 15 categories and 213,119 human-annotated visual question-answer pairs, together with TomaBench, an evaluation benchmark of multiple-choice and open-ended questions organized into a seven-task, three-level taxonomy. The authors evaluate 14 vision-language models and CLIP-based models in a zero-shot setting, report generally low accuracy on the benchmark, and introduce a fine-tuned model, TomaLLaVA, which reaches 96.09% accuracy on the MCQ portion. The central claim is that TomaBench exposes pronounced gaps in fine-grained recognition and factually grounded reasoning in current VLMs, and that simple fine-tuning on TomaMMU substantially narrows this gap.","tokens_in":16900,"tokens_out":6726,"duration_ms":60938,"significance":"If the benchmark's validity and the held-out status of TomaBench are established, TomaMMU would be a useful, large-scale resource for agricultural VQA: it is publicly released, combines field and in-house images, covers 14 diseases plus a healthy class with seven task types, and provides a systematic zero-shot comparison across multiple model families. The hierarchical taxonomy from Basic Perception to Expert Diagnosis is a reasonable structure for probing diagnostic reasoning, and the external zero-shot evaluations give the benchmark independent grounding. However, the manuscript's central quantitative claims currently rest on unresolved internal inconsistencies in the dataset counts and on the absence of a demonstrated split between fine-tuning data and TomaBench, so the resource's value cannot yet be assessed at face value.","major_comments":[{"comment":"The reported dataset sizes are mutually inconsistent. Section 3.1 states a training set of 124,367 VQA pairs and an evaluation set of 42,626 MCQs plus 37,816 OEQs called TomaBench; the Introduction and Conclusion describe TomaBench as 42,626 question-answer pairs; the dataset total is 213,119; and Fig. 1 shows a 60/20/20 split. The sum 124,367 + 42,626 + 37,816 = 204,809 differs from 213,119, and 60% of 213,119 is 127,871.4, not 124,367. Please provide a single consistent breakdown of how the 213,119 QA pairs are divided into training, validation, and TomaBench, and reconcile the MCQ and OEQ counts.","section":"§3.1, §3.2, Fig. 1, Introduction"},{"comment":"The central fine-tuning claim cannot be interpreted without an explicit leakage analysis. Stage 3 says only that 'a small, balanced subset, TomaBench, is selected for evaluation, while the remaining data is used for fine-tuning and training,' which is an assertion of intent rather than a demonstrated property. Because each MCQ and merged OEQ is templated from per-image metadata, a single shared or near-duplicate image between the training set and TomaBench would let TomaLLaVA memorize the associated metadata rather than perform diagnostic reasoning. The paper reports no image-level deduplication, no near-duplicate analysis, and no question/answer overlap check between the 124,367 fine-tuning pairs and TomaBench. Until this is established, the 96.09% MCQ result in §4.3 and Table 1 cannot support the claim that simple fine-tuning closes the gap.","section":"§3.2 (Stage 3), §4.3, Table 1"},{"comment":"The OEQ evaluation pipeline is internally inconsistent and is not validated. Section 4.1 says the LLM-as-judge uses 'the GPT score,' while Section 4.4 says 'Our prompt for Gemini 2.0 Flash is presented in Table S2'; Table 2 labels the column 'GPT'. The paper should name the judge model, state whether GPT or Gemini was used, and report human-agreement or correlation results for the 1-5 score. The approximate values (e.g., '∼2.3') are also not exact scores, and no confidence intervals are given for any of the Table 1 or Table 2 numbers, so small accuracy differences among models (e.g., 59.2 vs. 58.7) may not be meaningful.","section":"§4.1, §4.4, Table 2"},{"comment":"The benchmark's ground truth rests on expert annotation, but the manuscript provides no quantitative evidence of annotation quality. Section 3.2 states that 'a human domain expert strictly executes the entire metadata extraction and annotation pipeline' and that 'every data point undergoes a detailed manual verification process,' yet no inter-annotator agreement, error rate, or sample verification statistics are reported. Given that images are resized to 224×224 px, the claim that symptoms are 'visually identifiable' needs support; fine-grained symptoms such as small necrotic spots or mosaic patterns may not be reliably judged at this resolution. Please report annotation-quality measurements and discuss the resolution choice.","section":"§3.2 (Stage 2), §3.3, §3.1"}],"minor_comments":[{"comment":"The sentence 'Our results pronounced gaps...' is missing a verb; it should read 'Our results reveal pronounced gaps...'.","section":"Abstract"},{"comment":"The image count differs between text and figure: the text says 28,808 images, while Fig. 3a shows 28,799; also, Fig. 3a lists 'Bacterial Floundering' while Table S1 uses 'Bacterial Wilt'.","section":"§3.2, Fig. 3a"},{"comment":"Table 1 reports TomaLLaVA's average accuracy as 96.1 while the CSI column is omitted ('-'). The averaging convention is not defined; if the average is over the six reported tasks, state this explicitly, and if the CSI column is excluded from the average, say so.","section":"Table 1"},{"comment":"The text cites 'Fine-tuned VLMs (AgriCLIP, 93.73% Acc, 0.948 F1)' but AgriCLIP appears neither in Table 1 nor in the model list in Section 4; either include the full AgriCLIP evaluation or remove the comparison.","section":"§4.3"},{"comment":"Model names are inconsistent between tables: 'LLaVA-1.5-7B' in Table 1 versus 'LLaVA-OV' in Table 2 for reference [16]; clarify which checkpoint was evaluated.","section":"Tables 1 and 2"},{"comment":"No prompt template is provided for MCQ answering or for zero-shot evaluation of non-generative CLIP-based models; please include the exact prompts and the protocol used for models that do not natively generate answer options.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript would benefit from an explicit statement that TomaBench is a held-out set that is disjoint from the fine-tuning split, with the overlap analysis performed after any re-split. Given the unresolved count inconsistencies, a rerun of the fine-tuning comparison after defining the split correctly may be necessary. The authors' prior line of work on SCOLD and LeafNet is closely related to the components used in TomaLLaVA, so the relationship between the new benchmark and those earlier resources should be made transparent."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about TomaMMU: it is a genuinely useful-looking tomato-specific multimodal dataset, but the paper's headline result—that fine-tuning on TomaMMU pushes MCQ accuracy to 96.09%—is uninterpretable until the authors show that TomaBench is actually disjoint from the fine-tuning data. Right now they don't.\n\nWhat is new: the dataset artifact itself (28,808 images, 213,119 human-annotated QA pairs, seven task types under a three-level hierarchy), the released data and code, and a broad zero-shot evaluation of fourteen VLMs with a structured error analysis. That is real work, and the ambition of going beyond classification to diagnostic reasoning is worth taking seriously. The human annotation pipeline is described in enough detail to be replicable, and the HF release makes external checking possible.\n\nWhere it gets soft: the numbers do not add up. The abstract says 213,119 QA pairs; Section 3.1 says 124,367 training + 42,626 MCQs + 37,816 OEQs = 204,809, leaving over 8,000 pairs unexplained. The Introduction and Conclusion say TomaBench has 42,626 QA pairs, which contradicts the 42,626 MCQs plus 37,816 OEQs. Image count also wavers between 28,808 and 28,799 in Figure 3. These inconsistencies matter because the central claim depends on a clean split. Section 3.2 Stage 3 states that 'a small, balanced subset, TomaBench, is selected for evaluation' but provides no image-level deduplication, no near-duplicate check, and no QA overlap analysis. Since questions are templated from per-image metadata, a single near-duplicate image between training and test could let the model memorize metadata rather than demonstrate diagnosis. The gain pattern (96.1% vs. 61.3% zero-shot) is consistent with leakage. The paper also lacks inter-annotator agreement for the human labels and any reliability check for the GPT-score judge. And with 89.69% of images from the authors' own LeafNet, a quantitative comparison against LeafBench is needed to position the contribution.\n\nIs the central argument salvageable? Yes. The benchmark itself is plausible, the zero-shot underperformance finding is robust across many models, and the flaws are addressable. But the current version is a conditional accept at best.\n\nFor peer review: send it out. A serious referee can push for the leakage analysis and number corrections. This paper is for researchers building agricultural VQA benchmarks and practitioners wanting tomato disease models; they will get value from it once the evaluation is trustworthy.","headline":"Useful tomato VQA dataset and benchmark, but the 96.09% fine-tuning claim is unverifiable until the authors prove TomaBench is disjoint from the training split.","tokens_in":17461,"tokens_out":2459,"would_cite":false,"duration_ms":23377,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that current vision-language models, tested zero-shot on a new tomato-disease benchmark, fall short on fine-grained symptom recognition and diagnostic reasoning, while fine-tuning on the TomaMMU dataset lifts…","keywords":["Vision-Language Models","Multimodal Dataset","Visual Question Answering","Multimodal Benchmark","Agricultural decision support","Tomato leaf disease","Fine-grained recognition","TomaBench"],"falsifier":"Take a random sample of TomaBench images and have independent plant pathologists re-answer the questions, then compute inter-annotator agreement. Separately, check for image or question-template overlap between the fine-tuning set and TomaBench. If agreement is low or overlap is nonzero, the benchmark's central comparisons do not hold.","tokens_in":16332,"feed_emoji":"🍅","tokens_out":4999,"duration_ms":48945,"temperature":0.7,"pith_summary":"Vision-language models are being proposed for agricultural decision support, but the paper argues that no benchmark has measured whether they can actually diagnose tomato leaf disease beyond simple classification. TomaMMU supplies that measurement: 28,808 field and laboratory images, 213,119 human-annotated question-answer pairs, and a held-out TomaBench of 42,626 multiple-choice and 37,816 open-ended questions spanning seven tasks from basic perception to expert diagnosis. Testing 14 current VLMs zero-shot, the paper finds they underperform precisely where diagnosis is hardest—symptoms, pathogens, scientific names—and that simple fine-tuning on TomaMMU raises MCQ accuracy to 96.09%, far above every zero-shot baseline. If the benchmark is sound, it gives the field both a target to beat and evidence that domain-specific data, not model scale, is what moves agricultural VLM performance.","feed_headline":"Fine-tuning on 124K tomato-disease QAs lifts accuracy to 96%","feed_subtitle":"Zero-shot vision-language models score near chance on pathogen and symptom questions; domain fine-tuning closes the gap.","key_machinery":"The load-bearing object is TomaBench's hierarchical task structure: seven question types—Healthy/Diseased Classification, Leaf Count, Crop Species Identification, Symptom Identification, Pathogen Classification, Disease Classification, Scientific Name Classification—stacked into Basic Perception, Pathology Understanding, and Expert Diagnosis levels. The argument runs through this ladder: models climb from binary screening to fine-grained morphological reasoning, and the benchmark's score profile (strong on Perception, weak on Pathology/Expert) is what exposes the recognition-and-reasoning gap. The fine-tuned TomaLLaVA, built from SCOLD as image encoder and LFM2.5-1.2B-Instruct as base LLM with LoRA training, is the vehicle that demonstrates the gap closes with domain data.","core_discovery":"TomaMMU is a curated multimodal dataset of 28,808 tomato-leaf images (15 categories) with 213,119 human-annotated VQA pairs, and TomaBench is its held-out evaluation core of 42,626 MCQs and 37,816 OEQs organized into seven tasks under three levels of diagnostic depth. On TomaBench, the paper reports that 14 state-of-the-art vision-language models, evaluated zero-shot, suffer pronounced drops on fine-grained tasks (pathogen classification, symptom identification, scientific-name classification), with even the strongest proprietary model averaging only 70.3% on MCQs and roughly 2.5/5 on open-ended questions. Fine-tuning a compact model (SCOLD encoder + LFM2.5-1.2B base, LoRA) on the 124,367 training QA pairs yields 96.09% average MCQ accuracy, +34.8 points over the best zero-shot baseline, which the paper attributes to domain-specific supervision rather than scale.","pith_inferences":["A formal leakage check between TomaBench and the fine-tuning split is the immediate next test; the paper does not report one, so the 96.09% gain should be read as provisional.","Because answers were verified at 224x224 resolution, the benchmark may underestimate what higher-resolution models could see; re-annotation at native resolution could change difficulty.","The OEQ scores (~2.3 to 2.7 on a 5-point scale) suggest the MCQ results overstate real-world readiness; open-ended diagnosis, the format closer to farmer use, remains substantially harder.","The same three-stage pipeline could transfer to other crops, making TomaBench a template for crop-specific diagnostic benchmarks rather than a tomato-only resource."],"forward_implications":["Agricultural decision-support systems built on general VLMs should not be deployed for tomato disease diagnosis without domain adaptation, since the best zero-shot model averages only 70.3% on MCQs and near-random scores on symptom, pathogen, and scientific-name tasks.","TomaBench gives the field a reusable evaluation protocol with seven tasks across three reasoning levels, so future agri-VLM claims can be compared on the same ground.","Because fine-tuning a small model on TomaMMU surpasses much larger zero-shot models, domain-specific data collection may matter more than parameter count for agricultural expertise.","The documented error categories—perceptual error and refusal, core miss, and granularity omission—point developers to specific failure modes to target."],"supporting_citations":[{"why":"Supplies the majority of in-the-wild tomato leaf images that ground TomaMMU in real-world conditions.","marker":"[27]"},{"why":"Contributes the TOM2024 field images used as the second data source for the dataset.","marker":"[3]"},{"why":"Provides the seasoned three-category taxonomy (Basic Perception, Pathology Understanding, Expert Diagnosis) that shapes TomaBench's structure.","marker":"[32]"},{"why":"Serves as the SCOLD image encoder inside TomaLLaVA and as a domain-specific zero-shot baseline.","marker":"[28]"},{"why":"Supplies the LFM2.5-1.2B-Instruct base language model that TomaLLaVA fine-tunes, and a strong zero-shot comparison point.","marker":"[2]"},{"why":"Provides Gemini 2.0 Flash, the strongest proprietary zero-shot baseline and the LLM judge that grades open-ended responses.","marker":"[4]"},{"why":"Represents the prior crop-disease multimodal benchmark that TomaMMU extends in scale and reasoning depth.","marker":"[17]"},{"why":"Establishes the hierarchical agricultural benchmark tradition that TomaBench refines with seven task-specific question types.","marker":"[48]"}],"fun_headline_variants":["Fine-tuning on tomato QA lifts VLM accuracy to 96%, exposing zero-shot gaps","Zero-shot VLMs fail tomato disease diagnostics; fine-tuning reaches 96%","New tomato leaf benchmark shows fine-tuning beats zero-shot by 35 points","Domain fine-tuning closes VLM gap on tomato disease understanding to 96%","Tomato disease dataset and benchmark reveal VLM limits; fine-tuning succeeds"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire benchmark rests on the accuracy of its human-curated answers: if the ground-truth labels are wrong, or if any of the 42,626 evaluation questions overlap the 124,367 fine-tuning pairs, then both the measured performance gap and the fine-tuning gain are artifacts rather than facts about the models.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuning on tomato QA lifts VLM accuracy to 96%, exposing zero-shot gaps","Zero-shot VLMs fail tomato disease diagnostics; fine-tuning reaches 96%","New tomato leaf benchmark shows fine-tuning beats zero-shot by 35 points","Domain fine-tuning closes VLM gap on tomato disease understanding to 96%","Tomato disease dataset and benchmark reveal VLM limits; fine-tuning succeeds"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000922,"raw_usage":{"total_tokens":3987,"prompt_tokens":1013,"completion_tokens":2974,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":629,"completion_tokens_details":{"reasoning_tokens":2873}},"tokens_in":629,"tokens_out":2974,"duration_ms":22966,"temperature":1.0,"reasoning_tokens":2873,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:26:34.506307+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of TomaBench images and have independent plant pathologists re-answer the questions, then compute inter-annotator agreement. Separately, check for image or question-template overlap between the fine-tuning set and TomaBench. If agreement is low or overlap is nonzero, the benchmark's central comparisons do not hold.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the TOM2024 field images used as the second data source for the dataset."},{"cited_title":"PlantExpertVQA: A Visual Question Answering Dataset for Benchmarking Vision-Language Models in Plant Science","cited_arxiv_id":"2508.17117","evidence_quote":"Provides the seasoned three-category taxonomy (Basic Perception, Pathology Understanding, Expert Diagnosis) that shapes TomaBench's structure."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Serves as the SCOLD image encoder inside TomaLLaVA and as a domain-specific zero-shot baseline."},{"cited_title":"Frontiers in Artificial Intelligence8(Mar 2025)","cited_arxiv_id":null,"evidence_quote":"Provides Gemini 2.0 Flash, the strongest proprietary zero-shot baseline and the LLM judge that grades open-ended responses."}],"review_version":1}