{"id":"64b1f394-ce6f-4862-bc3f-1074ec83c2a2","arxiv_id":"2504.20419","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Fine-tuned GPT-4o slightly outperforms ResNet-50 on PlantVillage apple and corn disease classification (98.12% vs 96.88% at 256px) at much higher compute cost.","lead":"This paper compares OpenAI's GPT-4o with a standard ResNet-50 CNN for classifying apple and corn leaf diseases from PlantVillage images, after fine-tuning both on a few hundred labeled images. Fine-tuned GPT-4o reached up to 98.12% accuracy on apple leaves, slightly ahead of ResNet-50's 96.88%, but it cost hundreds of times more to train and ran about twenty times slower.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-split accuracy gaps as small as 2/160 test images support the GPT-4o superiority claim; no confidence intervals, multiple seeds, or equalized training sets are provided, so the central comparison is not yet statistically load-bearing.","rationale":"The paper's own reporting supports this concern: Table II lists single point estimates, Section III.B fixes one random state, and Sections IV.A and IV.C document non-repeatable content-filter removals. I did not raise the title's 'LLM+CNN' framing or the missing hyperparameter values as primary because neither is necessary for the falsifiable core comparison. I credit the authors for disclosing the filter issue and providing a public repository, but disclosure does not turn a two-image difference into evidence. The reader's CONDITIONAL verdict is appropriate; the proposed test would either strengthen the claim with error bars and equalized training sets or show that the current evidence is insufficient.","tokens_in":31176,"tokens_out":6404,"duration_ms":71472,"concrete_test":"Rerun the Apple-256px comparison (the flagship result) on 20 random splits of the 800-image balanced set, using identical random seeds and equalized training sets by applying the same content-filter exclusion mask to both GPT-4o and ResNet-50; report mean accuracy difference with a 95% bootstrap CI and a paired McNemar test per split. If the mean GPT-4o minus ResNet-50 advantage is not positive with a CI excluding zero, or does not hold in a majority of splits, the central superiority claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"For the central claim to hold, the Table II accuracy gaps must reflect a real and reproducible difference. The flagship result (Apple 256px) is 0.9812 vs 0.9688, i.e., 157 vs 155 correct out of 160 test images; a two-image change flips the comparison. With n=160, the approximate standard error of each accuracy is about 1.1 to 1.4 percentage points, so the 1.24-point gap is within sampling noise. Several other comparisons are similarly small (Apple 150: 5 images; Corn 256: 7 images). All results come from one split (random_state=42) with no confidence intervals or seed variation. The comparison is additionally confounded by OpenAI's content filter, which removed 7 apple and 55 corn images from the initial GPT training sets but different counts when the same datasets were used in progressive fine-tuning (52 apple, 61 corn), so the two models were trained on different data and the exact experiment cannot be repeated. The progressive fine-tuning comparison used to support hyperparameter transfer is also not controlled: it compares a 10-epoch full-training run with 3-epoch-per-phase runs that themselves lost up to 43 images to filtering. The paper is transparent about these limitations, but the claimed 'slightly better' performance and 'improved generalization' require more than point estimates.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript evaluates GPT-4o against ResNet-50 for four-class plant disease classification on balanced subsets of the PlantVillage dataset (apple and corn, at 100/150/256 px). It reports zero-shot, few-shot fine-tuned, and progressively fine-tuned performance, plus cross-resolution and cross-plant transfer experiments. The central claim is that fine-tuned GPT-4o slightly outperforms ResNet-50, reaching 98.12% accuracy on apple at 256 px versus 96.88% for ResNet-50, with improved generalization and near-zero training loss. The authors also claim that hyperparameters optimized via Bayesian optimization on ResNet-50 transfer effectively to GPT-4o fine-tuning, based on a comparison with a default-hyperparameter progressive training protocol.","tokens_in":31512,"tokens_out":3104,"duration_ms":35197,"significance":"If the central comparative claim were statistically supported, the paper would provide a useful data point on multimodal LLM fine-tuning for plant disease classification, including cost, speed, and resolution-sensitivity trade-offs. It would also offer a practical hyperparameter-transfer recipe (ResNet-50 Bayesian optimization followed by GPT-4o fine-tuning) that could reduce API experimentation costs. The paper is transparent about data-exclusion events caused by OpenAI's content filter, reports monetary costs and runtimes, and makes code/data publicly available. However, as presented, the flagship accuracy gaps are not supported beyond point estimates: the Apple 256 px comparison is 157 versus 155 correct test images out of 160, a difference within sampling noise, and all comparisons rest on a single fixed split with unequal training sets due to non-reproducible content-filter removals. These issues are load-bearing for the abstract's and Section V.A's comparative conclusions.","major_comments":[{"comment":"The headline claim that fine-tuned GPT-4o 'slightly outperforms' ResNet-50 is not statistically supported. With n=160 test images, the approximate standard error of each accuracy is about 1.1–1.4 percentage points, and the Apple 256 px gap of 1.24 points corresponds to only two test images (157 vs. 155 correct). Several other gaps are similarly small in count terms (Apple 150: 5 images; Corn 256: 7 images). All numbers come from a single split with random_state=42; no confidence intervals, standard errors, bootstrap estimates, or repeated-seed evaluations are provided. Because the abstract and Section V.A draw the central comparative conclusion from these point estimates, the paper needs at least a proper uncertainty analysis or multiple-seed evidence before claiming a real difference in accuracy or 'improved generalization.'","section":"IV.B, Table II; III.B"},{"comment":"The GPT-4o and ResNet-50 training sets were not equal, and the GPT-4o training sets differed across configurations and across repeated runs of the same data. Section IV.A reports that OpenAI's content filter removed 7 apple and 55 corn images from the initial GPT-4o fine-tuning sets; Section IV.C reports different counts for the same datasets (52 apple, 61 corn) in progressive fine-tuning, with per-phase exclusions of 0–43 and 0–14 images. The models are therefore compared after training on different subsets, and the authors themselves attribute the Corn 150 px accuracy drop to the exclusion of 35 images. Under these conditions, a clean model comparison is not possible, and the cross-resolution patterning in Table II may reflect data filtering rather than model capability. The manuscript should either equalize training sets by removing the same flagged images from all models, or present the comparison under matched subsets as a sensitivity analysis.","section":"IV.A, IV.C, Table III"},{"comment":"The claim that ResNet-50-derived hyperparameters 'transfer' to GPT-4o is supported by a comparison that selects the best progressive-fine-tuning phase on the same test set used for evaluation. Table A1 reports accuracy for phases 1–4 at each resolution, and Section V.B then compares the best phase (e.g., Apple 256 phase 3, 95.63%) with the full-training result (98.12%) to conclude a 2.49% advantage for Bayesian-optimized hyperparameters. Choosing the phase with highest test accuracy is a form of test-set model selection, which inflates the apparent gap and invalidates the comparison as an unbiased estimate. Furthermore, the progressive runs use 3-epoch default hyperparameters and only 128-sample increments, so the comparison conflates hyperparameter choice with data quantity and training protocol. Phase selection should be done on validation data, or a nested evaluation should be reported.","section":"IV.C, V.B, Table A1"},{"comment":"The claim of 'improved generalization' is not established by the reported evidence. Section IV.A infers generalization from lower training/validation loss in Table I, but near-zero training loss is more naturally read as sign of capacity or memorization, and Table II test accuracies do not systematically align with the validation-loss ordering. The abstract and Section V.A present the generalization claim as a finding, while Section V.D later states that the study 'does not aim to establish whether GPT-4o outperforms other models.' The manuscript should either present test-based evidence for generalization (e.g., held-out robustness metrics, repeated-split generalization intervals) or soften the abstract and conclusion accordingly.","section":"IV.A, V.A, V.D"}],"minor_comments":[{"comment":"The table title contains a typo, 'RENNET-50', and the training-duration cell '247.2.0' contains an extra decimal point.","section":"Table I"},{"comment":"Several F1 entries are shown with three decimals (e.g., 0.956, 0.772, 0.881) while others have four; the table would be clearer with consistent formatting.","section":"Table A1"},{"comment":"Model naming is inconsistent: 'GPT-4o', 'gpt-4o', and 'GPT-Resolution-256-Corn' are used in different places; a single naming convention would improve reproducibility.","section":"Throughout"},{"comment":"The sentence in Section V.D that the study does not aim to establish whether GPT-4o outperforms other models directly conflicts with the abstract's comparative claim and with Section V.A's interpretation of the accuracy gaps; this should be reconciled.","section":"V.D"},{"comment":"The term 'few-shot learning' is used for fine-tuning on 512 labeled images per crop, which is considerably more than typical few-shot conventions; the manuscript should clarify the sample-size terminology or use 'small-sample fine-tuning.'","section":"III.A, III.E"},{"comment":"Some references are duplicated (e.g., [1] and [14] and related entries), and several citation numbers in the text are not in sequential order; a reference-management pass would help.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable empirical study with useful cost and runtime data, but the central comparative claim currently rests on single-split, unequal-training-set accuracy point estimates. I do not see a fundamental irreparability issue: adding confidence intervals, multiple seeds or an alternative matched-subset comparison, and fixing the phase-selection protocol would address the main concerns. The authors' transparency about the content-filter exclusions is a strength, but it also shows that the exact experiment cannot be repeated as described, so the revision should explicitly bound the effect of those exclusions on the conclusions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's what I think. The genuinely new bit is the head-to-head: fine-tuned GPT-4o vs ResNet-50 on the same PlantVillage subsets across three resolutions, plus the observation that ResNet-derived hyperparameters transfer to GPT-4o fine-tuning without extra tuning. That's a legitimate empirical contribution, and the authors release code and data. I also give them credit for being straight about the OpenAI content-filter mess: they document how many images got removed in each run, notice the numbers differ between runs, and flag the reproducibility problem themselves. That is not a group trying to hide things.\n\nThe soft spot is exactly where the stress-test lands. The flagship result is 98.12% vs 96.88% on apple at 256px, which is 157 vs 155 correct out of 160. A two-image swing flips the comparison. With no confidence intervals, no multiple seeds, and a single fixed split, those gaps are sampling noise. Even the bigger gaps (10–12 points at 100px) could shift with seed variation, and the two models were trained on different image sets because the filter removed 7 and 55 images from the first GPT runs and different counts in progressive fine-tuning. So the central \"slightly better\" claim is not yet statistically load-bearing.\n\nThe hyperparameter-transfer argument is also weaker than it looks. It compares a 10-epoch full-training run against phase 3 of a progressive routine that used OpenAI defaults, a different batch size, and lost images to filtering. That's not a controlled test of \"same hyperparameters work.\" The paper even hedges in the discussion, saying it does not aim to establish superiority — which undercuts the abstract. There is a real tension there.\n\nWho should read it? Practitioners in agricultural AI who want a realistic cost-accuracy trade-off snapshot for one particular GPT-4o version. It's a useful benchmark report, not a convincing comparative result.\n\nRecommendation: send it to peer review, but ask for multiple seeds, confidence intervals, equalized training sets, and the actual hyperparameter values. If the gaps survive that, the paper becomes valuable. If not, it is still a reasonable engineering report.","headline":"A genuinely useful benchmark report that is transparent about its own limitations, but the headline GPT-4o superiority claim rests on single-split point estimates too small to carry it.","tokens_in":32008,"tokens_out":2663,"would_cite":false,"duration_ms":26807,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuned GPT-4o reaches 98.12% accuracy on apple leaf disease classification, beating ResNet-50 on the same test set.","keywords":["plant disease detection","multimodal large language models","GPT-4o","few-shot fine-tuning","ResNet-50","zero-shot classification","image resolution generalization","precision agriculture"],"falsifier":"Run the same few-shot fine-tuning protocol on multiple random splits while logging which training images the content filter removes each time; if GPT-4o's 98.12% accuracy or its edge over ResNet-50 on the 256-pixel apple test set does not survive across splits, the headline result is tied to one particular split rather than to the model.","tokens_in":31017,"feed_emoji":"🍎","tokens_out":10714,"duration_ms":103075,"temperature":0.7,"pith_summary":"This paper tries to show that a multimodal large language model — GPT-4o, which can look at images as well as read text — can be turned into an accurate plant-disease classifier with only a small amount of labeled leaf images. On a public benchmark of apple and corn leaf images, the authors fine-tuned GPT-4o on 512 images per crop and compared it with a standard convolutional network, ResNet-50. The fine-tuned GPT-4o reached 98.12% accuracy on apple leaves at 256 pixels, ahead of ResNet-50's 96.88%, and its advantage widened at lower resolutions, reaching 12.5 points on corn at 100 pixels. Zero-shot classification was far weaker, so the training matters; cross-plant transfer was poor, meaning the model does not generalize from one crop to another. A sympathetic reader would take the paper's contribution to be evidence that few-shot fine-tuned multimodal LLMs are a viable alternative to CNNs for this task, and that CNN-tuned hyperparameters can be reused to make such tuning affordable.","feed_headline":"GPT-4o tops ResNet-50 on apple leaf disease: 98.12% accuracy","feed_subtitle":"Trained on just 512 labeled images, the vision-language model edges out the CNN and holds up better at low resolution.","key_machinery":"The argument runs on three linked mechanisms. First, the dataset is balanced by undersampling each crop to 800 images, split 80/20 with a fixed random state of 42 into 512 training, 128 validation, and 160 test images, with equal class counts in every subset. Second, Bayesian optimization (30 trials with early stopping and pruning) finds ResNet-50's hyperparameters; reusing those same hyperparameters for GPT-4o fine-tuning is what makes the comparison and the cost argument work, because a full GPT-4o hyperparameter search would be prohibitively expensive. Third, the zero-shot and few-shot comparisons use a JSON-formatted prompt that tells the model to output one of four disease labels for the uploaded image. The progressive fine-tuning runs, which add 128 samples at a time, act as a check on whether the transferred hyperparameters or the platform's defaults are responsible for the result, and the paper finds that the transferred hyperparameters give the better accuracy.","core_discovery":"The central claim is that fine-tuned GPT-4o outperforms ResNet-50 in few-shot plant disease classification across all tested resolutions and both crops, with accuracy up to 98.12% on apple leaves (96.88% for ResNet-50 at 256 pixels) and 93.13% on corn at 100 pixels. The authors also claim that hyperparameters optimized for ResNet-50 via Bayesian search transfer cleanly to GPT-4o fine-tuning; this transferred setting beat the API platform's default settings by up to 2.49% accuracy while training loss reached zero and validation loss stayed low. The paper's own results show the limits: zero-shot GPT-4o reaches only 56.87% on apple and 69.37% on corn, models trained at 100 pixels generalize better to 256-pixel inputs than the reverse, and cross-plant accuracy tops out at 66.87%. During fine-tuning, the API platform's automatic content filter removed 3 to 35 training images per run and flagged different images on repeat runs, which the authors identify as an uncontrolled source of variation in their accuracy numbers.","pith_inferences":["Editorial extension: because the paper reports one fixed split, the 1.24-point top-line gap is a point estimate; rerunning across several random splits would show whether GPT-4o's advantage is stable or within run-to-run noise.","Editorial extension: the content filter removed different image sets on repeated fine-tuning runs, so even identical submitted jobs may train on different data; logging the IDs of filtered images would let others reproduce or adjust for this.","Editorial extension: the prompt asks only for a class label; adding a one-sentence textual description of each disease to the prompt would test whether the model's language knowledge can reduce the number of images needed."],"forward_implications":["A deployer could classify apple-leaf diseases at near-98% accuracy with 512 labeled images per crop, without collecting large annotation sets.","The reusable hyperparameters mean practitioners can tune a small CNN to choose settings for an LLM, avoiding a costly LLM-specific search.","Low-resolution training followed by high-resolution inference gave the best cross-resolution results, suggesting that field systems may not need high-resolution sensors to train on.","Zero-shot performance is too weak for use without fine-tuning, so an out-of-the-box general-purpose model is not yet a drop-in detector.","Cross-plant accuracy stays below 67%, so a model fine-tuned on one crop should not be expected to diagnose another crop's diseases."],"supporting_citations":[{"why":"Supplies the leaf-image dataset and its per-disease folders that all fine-tuning and test sets are drawn from.","marker":"[79]"},{"why":"Establishes the dataset's origin as a deep-learning plant-disease benchmark and provides the manual label annotations.","marker":"[80]"},{"why":"Supplies the premise that vision fine-tuning on around a hundred images can improve GPT-4o, motivating the few-shot setup.","marker":"[86]"},{"why":"Documents ResNet-50 as a strong standard baseline for plant disease classification, which the paper compares against.","marker":"[84, 85]"},{"why":"Provides the prompt-generation tool used to settle on the JSON prompt format for image classification.","marker":"[89]"},{"why":"Supplies the Bayesian-optimization method used to tune ResNet-50 hyperparameters that are then transferred to GPT-4o.","marker":"[94]"},{"why":"Gives the prior cross-crop binary-classification result that the paper cites to contextualize its four-class cross-plant failure.","marker":"[96]"}],"fun_headline_variants":["Fine-tuned GPT-4o beats ResNet-50 on plant disease: 98.12%","GPT-4o with few-shot tuning outperforms CNN on leaf disease","Zero-shot GPT-4o fails, but fine-tuning yields 98% accuracy on apples","Multimodal LLM outperforms CNN on plant disease after fine-tuning","GPT-4o fine-tuned on 512 images hits 98% accuracy on apple disease"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that a single fixed 80/20 split — 160 test images per crop — is representative enough that the accuracy gaps between GPT-4o and ResNet-50 reflect genuine model differences, and that the images silently dropped by the automatic content filter did not bias the comparison.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuned GPT-4o beats ResNet-50 on plant disease: 98.12%","GPT-4o with few-shot tuning outperforms CNN on leaf disease","Zero-shot GPT-4o fails, but fine-tuning yields 98% accuracy on apples","Multimodal LLM outperforms CNN on plant disease after fine-tuning","GPT-4o fine-tuned on 512 images hits 98% accuracy on apple disease"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000723,"raw_usage":{"total_tokens":3305,"prompt_tokens":1067,"completion_tokens":2238,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":683,"completion_tokens_details":{"reasoning_tokens":2126}},"tokens_in":683,"tokens_out":2238,"duration_ms":16760,"temperature":1.0,"reasoning_tokens":2126,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:29:17.693859+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same few-shot fine-tuning protocol on multiple random splits while logging which training images the content filter removes each time; if GPT-4o's 98.12% accuracy or its edge over ResNet-50 on the 256-pixel apple test set does not survive across splits, the headline result is tied to one particular split rather than to the model.","supporting_citations":[{"cited_title":"GitHub - spMohanty/PlantVillage-Dataset: Dataset of diseased plant leaf images and corresponding labels","cited_arxiv_id":null,"evidence_quote":"Supplies the leaf-image dataset and its per-disease folders that all fine-tuning and test sets are drawn from."},{"cited_title":"Introducing vision to the fine-tuning API | OpenAI","cited_arxiv_id":null,"evidence_quote":"Supplies the premise that vision fine-tuning on around a hundred images can improve GPT-4o, motivating the few-shot setup."},{"cited_title":"Automatically generate first draft prompt templates - Anthropic","cited_arxiv_id":null,"evidence_quote":"Provides the prompt-generation tool used to settle on the JSON prompt format for image classification."},{"cited_title":"A Comparative Analysis of Pretrained Models for Brain Tumaor Classification and Their Optimization Using Optuna,","cited_arxiv_id":null,"evidence_quote":"Supplies the Bayesian-optimization method used to tune ResNet-50 hyperparameters that are then transferred to GPT-4o."},{"cited_title":"Innovative deep learning approach for cross -crop plant disease detection: A generalized method for identifying unhealthy leaves,","cited_arxiv_id":null,"evidence_quote":"Gives the prior cross-crop binary-classification result that the paper cites to contextualize its four-class cross-plant failure."}],"review_version":1}