Pith. sign in

REVIEW 4 major objections 3 minor 1 cited by

PlantExpertVQA: A Visual Question Answering Dataset for Benchmarking Vision-Language Models in Plant Science

T0 review · 4 major / 3 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper introduces a 765,186-question visual QA dataset for plant disease diagnosis and reports that frontier vision-language models underperform on it while small-model fine-tuning sharply improves accuracy.

desk verdict A potentially useful agricultural VQA resource whose central claim—that answering requires visual grounding—is untested because the QA pipeline derives from metadata and no leakage controls are reported. read the letter →

arxiv 2508.17117 v3 pith:V3BNYGLN submitted 2025-08-23 cs.CV cs.AIcs.LG

Syed Nazmus Sakib , Nafiul Haque , Mohammad Zabed Hossain , Shifat E. Arman This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords visualquestionansweringplantdiseasediagnosisvision-languagemodelsbenchmarkdatasetdomainadaptationparameter-efficientfine-tuningagriculturalAIimagemetadata
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that plant-disease diagnosis can be benchmarked as a visual question answering task rather than only as image classification, and that the field currently lacks a dataset large enough to train or evaluate models for interactive reasoning about crops and diseases. It introduces PlantExpertVQA, built from 45 open-source datasets, with 765,186 QA pairs over 150,841 images spanning 38 crop species and 89 disease conditions, organized into three cognitive-complexity levels and nine question categories. The paper reports that current frontier vision-language models score poorly on the benchmark, while parameter-efficient fine-tuning of a 2B-parameter model on a small fraction of the data improves accuracy across all categories. A sympathetic reading is that the dataset closes a gap between classification benchmarks and the interactive, reasoning-based questions agricultural advisers actually ask.

What carries the argument

The load-bearing object is the dataset itself, produced by a two-stage automated pipeline: template-based QA generation from image metadata, then multi-stage linguistic re-engineering, with domain-expert review for scientific accuracy. The question taxonomy, with three cognitive levels and nine question categories, is the measurement apparatus that lets the paper report per-category failures and show that fine-tuning improves all categories rather than only simple ones.

What would settle it

Run each benchmark question without the image, passing only the question text to a strong language model. If accuracy stays near the vision-language model's score, the questions do not test visual understanding, and the fine-tuning gains could come from learning answer patterns rather than from image reasoning.

Watch

Extended reading notes

Core claim

The paper's central claim is that PlantExpertVQA is a valid, large-scale benchmark for vision-language plant diagnosis: 765,186 QA pairs grounded in 150,841 images covering 38 crops and 89 disease conditions, organized into three cognitive-complexity levels and nine question categories, built by template-based synthesis from image metadata followed by multi-stage linguistic re-engineering and expert review. It further claims that current frontier vision-language models perform poorly on this benchmark, and that parameter-efficient fine-tuning of a compact 2B-parameter model on a small fraction of the dataset yields substantial gains across all question categories, evidence that the dataset captures meaningful domain knowledge.

Load-bearing premise

The load-bearing premise is that questions created from image metadata actually require looking at the image to answer, so that low scores and fine-tuning gains reflect visual reasoning rather than text patterns or leaked metadata.

Editorial extensions

If this is right

  • PlantExpertVQA gives the plant-science community a common testbed: future vision-language models can be compared on the same 765,186 questions, with per-category scores showing where visual reasoning still fails.
  • The reported fine-tuning result implies that a 2B-parameter model can absorb substantial plant-diagnosis capability from a small training slice, which is a practical path for deployment where large models are too costly.
  • The three-level cognitive taxonomy lets users identify which question types are hardest, pointing data collection toward reasoning skills rather than raw classification.
  • Distilling 45 existing datasets into one QA format makes the underlying images easier to reuse for training interactive diagnosis tools, not just classifiers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A text-only control would decide the benchmark's validity: if a language model given only the question and no image still scores near the vision-language baseline, the benchmark would be measuring template patterns rather than visual reasoning.
  • The same template-plus-rephrasing pipeline could be ported to other expert domains, such as insect damage or soil disorders, making the construction method a reusable template for domain VQA benchmarks.
  • Because the dataset aggregates existing image collections, checking for near-duplicate images across train and test splits would show whether the fine-tuning gains are partly inflated by image overlap.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper presents PlantExpertVQA, a large-scale visual question answering dataset for plant science, constructed from 45 open-source datasets and comprising 765,186 QA pairs over 150,841 images, 38 crop species, and 89 disease conditions. Questions are organized into 3 cognitive complexity levels and 9 categories, generated by a two-stage pipeline that synthesizes templates from image metadata and then performs linguistic re-engineering, followed by expert review. The paper evaluates several frontier vision-language models zero-shot, reports poor performance, and shows that parameter-efficient fine-tuning of a 2B-parameter model on a small fraction of the dataset yields improvements across all question categories. The central claim is that PlantExpertVQA is a valid benchmark for vision-language plant diagnosis and domain adaptation.

Significance. If the validity concerns are addressed, the dataset would be a useful resource: its scale and taxonomic breadth exceed existing plant-disease VQA efforts, the three-level cognitive complexity schema is a sensible design choice, and the expert-review step is a serious attempt at quality control. The demonstration that a compact 2B model can be domain-adapted is also practically relevant. However, the dataset's validity currently rests on an untested assumption that the synthesized questions require visual grounding, and several reporting gaps prevent independent verification. The contribution is therefore conditional on the proposed controls and on release of the artifact.

major comments (4)
  1. [Abstract (two-stage pipeline description)] The construction pipeline generates QA pairs from image metadata, but the paper reports no control condition that removes or degrades the image: no text-only baseline receiving the question plus crop/disease/source metadata, no image-blurring or image-shuffling ablation, and no analysis of whether answers are recoverable from question templates alone. Without such controls, low zero-shot accuracy and post-fine-tuning gains could reflect learning template-to-metadata mappings rather than visual reasoning, which would invalidate the central claim that the benchmark measures vision-language plant diagnosis. Please add these baselines and report per-category results.
  2. [Dataset availability (entire manuscript)] The paper provides no dataset URL, DOI, or code repository, yet it is a dataset and benchmark paper; the abstract's 765,186 QA pairs and 150,841 images cannot be inspected, and the exact split sizes, metadata schema, and template inventory cannot be checked. A benchmark paper must release the artifact and the generation and evaluation scripts to make the central claims reproducible.
  3. [Experiments (fine-tuning results)] The reported fine-tuning improvements are aggregate numbers with no error bars, no multiple-seed variance, and no specification of the exact 'small fraction' of data used; additionally, because multiple QA pairs are generated per image, the train/test split must be verified to have no image-level overlap to avoid optimistic estimates from same-image leakage. Please report mean and standard deviation over at least three seeds, the exact training set size and percentage, and the image-level split protocol.
  4. [Expert review and quality control] The abstract describes the dataset as 'iteratively reviewed by domain experts for scientific accuracy and relevance,' but no quantitative quality evidence is reported: the number of experts, the number and sampling of QA pairs reviewed, inter-annotator agreement (e.g., Cohen's or Fleiss' kappa), and the fraction of QA pairs discarded or rewritten are all absent. These numbers are necessary to support the 'high-quality' claim and to rule out systematic errors in template-generated questions.
minor comments (3)
  1. [Full text / formatting] Large portions of the supplied full text are unreadable due to encoding corruption, including parts of the tables and appendix, so I could not verify all figure and table numbering or the exact template examples; please provide a clean copy for the next round.
  2. [Abstract] The phrase 'a small fraction of the dataset' should be quantified in the abstract or at least in the experiments section; the exact number of training samples matters for judging the efficiency claim.
  3. [Dataset statistics] The paper should state the number of QA pairs per image, the distribution of QA pairs across the 9 categories and 3 complexity levels, and the degree of template duplication, since these affect both the leakage risk and the interpretation of accuracy scores.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the dataset construction and model evaluations are empirical, not derived from the claims they support.

full rationale

The paper's central claims are that PlantExpertVQA is a large benchmark and that frontier vision-language models score poorly on it while fine-tuning improves performance. These are empirical findings about a constructed dataset, not predictions derived from a fitted model or from a self-citation chain. The abstract explicitly describes the construction as 'template-based QA synthesis from image metadata, followed by multi-stage linguistic re-engineering,' and the evaluation is a standard train/held-out benchmark exercise. No equation or definition equates the claimed benchmark validity to the answer-generation procedure, and no parameter is fitted to a subset and then renamed as a prediction. The concern that questions may be answerable from metadata or question-template patterns is a dataset-validity risk, not a circularity: it would make the benchmark less diagnostic of vision-language ability, but it does not make the paper's reported accuracy numbers or fine-tuning gains true by construction. Since no load-bearing step reduces to its own input, the honest finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No new physical or conceptual entities are introduced; the contribution is an assembled dataset and benchmark. The load-bearing assumptions are about source-data quality, the visual nature of the generated questions, and the effectiveness of expert review.

assumptions (3)
  • domain assumption Labels and metadata in the 45 source datasets are accurate and consistently formatted.
    The QA pairs are synthesized from these metadata; errors propagate into the benchmark. This is assumed in the abstract's description of compiling the dataset from open-source corpora.
  • domain assumption Template-based QA synthesis produces questions that require visual analysis of the image rather than recall of metadata.
    This is the load-bearing premise for interpreting poor VLM performance as a vision gap; the abstract does not test for metadata leakage.
  • domain assumption Expert review ensures scientific accuracy and relevance of the generated QA pairs.
    The abstract states that the dataset was iteratively reviewed by domain experts, but no protocol, sample size, or agreement metrics are reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PlantExpertVQA: A Visual Question Answering Dataset for Benchmarking Vision-Language Models in Plant Science." pith.science (2026). https://pith.science/paper/V3BNYGLN

@misc{pith2026250817117,
  author       = {Pith},
  title        = {Pith review of: PlantExpertVQA: A Visual Question Answering Dataset for Benchmarking Vision-Language Models in Plant Science},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V3BNYGLN}},
  note         = {Machine review of arXiv:2508.17117}
}
read the original abstract

Existing plant-disease datasets target classification and detection, leaving vision-language models unable to support interactive, reasoning-based diagnosis. To address this, we present PlantExpertVQA, a large-scale visual question answering (VQA) dataset designed to advance vision-language models for agricultural decision-making. It is compiled from 45 open-source datasets, including the widely used PlantVillage corpus, and comprises 765,186 high-quality question-answer (QA) pairs grounded over 150,841 images spanning 38 crop species and 89 disease conditions. Questions are organized into 3 levels of cognitive complexity and 9 distinct categories. Each was phrased following expert guidance and generated via an automated two-stage pipeline: template-based QA synthesis from image metadata, followed by multi-stage linguistic re-engineering. The dataset was iteratively reviewed by domain experts for scientific accuracy and relevance. We find that current frontier vision-language models, including recent open-source instruction-tuned multimodal LLMs, perform poorly on PlantExpertVQA. However, parameter-efficient fine-tuning of a compact 2B-parameter model on a small fraction of the dataset yields substantial improvements across all question categories, demonstrating its effectiveness for domain adaptation.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. TomaMMU: A Comprehensive Multimodal Understanding Benchmark for Tomato Leaf Diseases

    cs.CV 2026-08 conditional novelty 4.0 of 10

    TomaMMU is a tomato-leaf-disease VQA dataset with more than 200,000 question-answer pairs, and TomaBench shows that current vision-language models underperform on it, while fine-tuning on TomaMMU lifts MCQ accuracy to...

Reference graph

Works this paper leans on

32 extracted references · 25 canonical work pages · cited by 1 Pith paper

  1. [1]

    BananaSqueezeNet: A very fast, lightweight convolutional neural network for the diagnosis of three prominent banana leaf diseases

    Md Abdullahil Baki Bhuiyan, Hasan Muhammad Abdullah, Shifat E Arman, Sayed Saminur Rahman, and Kaies Al Mahmud. BananaSqueezeNet: A very fast, lightweight convolutional neural network for the diagnosis of three prominent banana leaf diseases. Smart Agricultural Technology, 4:100214, 2023

    work page 2023

  2. [2]

    Deep learning for mango leaf disease identification: A vision transformer perspective

    Md Arban Hossain, Saadman Sakib, Hasan Muhammad Abdullah, and Shifat E Arman. Deep learning for mango leaf disease identification: A vision transformer perspective. Heliyon, 10(17), 2024

    work page 2024

  3. [3]

    The global burden of pathogens and pests on major food crops

    Serge Savary, Laetitia Willocquet, Sarah Jane Pethybridge, Paul Esker, Neil McRoberts, and Andy Nelson. The global burden of pathogens and pests on major food crops. Nature Ecology & Evolution, 3(3):430--439, 2019

    work page 2019

  4. [4]

    Mohanty, David P

    Sharada P. Mohanty, David P. Hughes, and Marcel Salath \'e . Using deep learning for image-based plant disease detection. Frontiers in Plant Science, 7:215232, 2016

    work page 2016

  5. [5]

    Ferentinos

    Konstantinos P. Ferentinos. Deep learning models for plant disease detection and diagnosis. Computers and Electronics in Agriculture, 145:311--318, 2018

    work page 2018

  6. [6]

    Lawrence Zitnick, and Devi Parikh

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2425--2433, 2015

    work page 2015

  7. [7]

    PMC-VQA: Visual instruction tuning for medical visual question answering

    Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. PMC-VQA: Visual instruction tuning for medical visual question answering. arXiv preprint arXiv:2305.10415, 2023

    arXiv 2023

  8. [8]

    PathVQA: 30000+ questions for medical visual question answering

    Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. PathVQA: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003.10286, 2020

    arXiv 2003

Show all 32 references
  1. [9]

    Stacked attention networks for image question answering

    Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked attention networks for image question answering. arXiv preprint arXiv:1511.02274, 2015

    2015 arXiv

  2. [10]

    Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering

    Yash Goyal, Tushar Khot, Daniel Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. arXiv preprint arXiv:1612.00837, 2017

    2017 arXiv

  3. [11]

    SLAKE: A semantically-labeled knowledge-enhanced dataset for medical visual question answering

    Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. SLAKE: A semantically-labeled knowledge-enhanced dataset for medical visual question answering. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 1650--1654. IEEE, 2021

    2021

  4. [12]

    Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3558--3568, 2021

    2021

  5. [13]

    MUTAN: Multimodal Tucker Fusion for Visual Question Answering

    Hedi Ben-Younes, R \'e mi Cadene, Matthieu Cord, and Nicolas Thome. MUTAN: Multimodal Tucker Fusion for Visual Question Answering. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2612--2620, 2017

    2017

  6. [14]

    Hudson and Christopher D

    Drew A. Hudson and Christopher D. Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6700--6709, 2019

    2019

  7. [15]

    Banerjee and S

    S. Banerjee and S. Bhattacharya. VQA for Education: A Survey of Recent Trends and Future Directions. arXiv preprint arXiv:2210.12345, 2022

    2022 arXiv

  8. [16]

    Huang and Z

    Y. Huang and Z. Wang. AI Tutors in the Classroom: A VQA-based Approach. Journal of Educational Technology, 48(2):112--128, 2022

    2022

  9. [17]

    Jain and R

    A. Jain and R. Gupta. A VQA-based System for Customer Support in E-commerce. In Proceedings of the International Conference on Information Systems (ICIS), pages 1--10, 2021

    2021

  10. [18]

    Zhu and X

    Q. Zhu and X. Li. Enhancing E-commerce Experience with Visual Question Answering. Journal of Retail and E-commerce, 12(3):45--62, 2023

    2023

  11. [19]

    LingoQA: Visual question answering for autonomous driving

    Ana-Maria Marcu, Long Chen, Jan H \"u nermann, Alice Karnsund, Benoit Hanotte, Prajwal Chidananda, Saurabh Nair, Vijay Badrinarayanan, Alex Kendall, Jamie Shotton, and others. LingoQA: Visual question answering for autonomous driving. In European Conference on Computer Vision ...

    2024

  12. [20]

    Yan and Y

    K. Yan and Y. Wang. RadVQA: A Visual Question Answering Benchmark for Radiology. Journal of Medical Imaging, 10(1):014501, 2023

    2023

  13. [21]

    PlantDoc: A dataset for visual plant disease detection

    Davinder Singh, Naman Jain, Pranjali Jain, Pratik Kayal, Sudhakar Kumawat, and Nipun Batra. PlantDoc: A dataset for visual plant disease detection. In Proceedings of the 7th ACM IKDD CoDS and 25th COMAD, pages 249--253, 2020

    2020

  14. [22]

    Accurate and versatile 3D segmentation of plant tissues at cellular resolution

    Adrian Wolny, Lorenzo Cerrone, Athul Vijayan, Rachele Tofanelli, Amaya Vilches Barro, Marion Louveaux, Christian Wenzl, S \"o ren Strauss, David Wilson-S \'a nchez, Rena Lymbouridou, and others. Accurate and versatile 3D segmentation of plant tissues at cellular resolution. eL...

    2020

  15. [23]

    AgroGPT: Efficient agricultural vision-language model with expert tuning

    Muhammad Awais, Ali Husain Salem Abdulla Alharthi, Amandeep Kumar, Hisham Cholakkal, Rao Muhammad Anwer, and others. AgroGPT: Efficient agricultural vision-language model with expert tuning. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 56...

    2025

  16. [24]

    LLaVA-PlantDiag: Integrating Large-scale Vision-Language Abilities for Conversational Plant Pathology Diagnosis

    Karun Sharma, Vidushee Vats, Abhinendra Singh, Rahul Sahani, Deepak Rai, and Ashok Sharma. LLaVA-PlantDiag: Integrating Large-scale Vision-Language Abilities for Conversational Plant Pathology Diagnosis. In 2024 International Joint Conference on Neural Networks (IJCNN), pages ...

    2024

  17. [25]

    Informed-Learning-Guided Visual Question Answering Model of Crop Disease

    Yunpeng Zhao, Shansong Wang, Qingtian Zeng, Weijian Ni, Hua Duan, Nengfu Xie, and Fengjin Xiao. Informed-Learning-Guided Visual Question Answering Model of Crop Disease. Plant Phenomics, 2024, 2024

    2024

  18. [26]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and others. Learning transferable visual models from natural language supervision. In International Conference on Machine Learnin...

    2021

  19. [27]

    LXMERT: Learning cross-modality encoder representations from transformers

    Hao Tan and Mohit Bansal. LXMERT: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv:1908.07490, 2019

    1908 arXiv

  20. [28]

    An open access repository of images on plant health to enable the development of mobile disease diagnostics

    David Hughes, Marcel Salath \'e , and others. An open access repository of images on plant health to enable the development of mobile disease diagnostics. arXiv preprint arXiv:1511.08060, 2015

    2015 arXiv

  21. [29]

    FLAVA: A foundational language and vision alignment model

    Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. FLAVA: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1...

    2022

  22. [30]

    Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman

    Jason J. Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images. Scientific Data, 5(1):1--10, 2018

    2018

  23. [31]

    George N. Agrios. Plant Pathology. Elsevier, 2005

    2005

  24. [32]

    Strange and Peter R

    Richard N. Strange and Peter R. Scott. Plant disease: a threat to global food security. Annual Review of Phytopathology, 43(1):83--116, 2005

    2005

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.