{"id":"bd9428a9-cfca-40f0-83be-e773e2ffb587","arxiv_id":"2411.15203","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A fine-tuned multimodal language model for wheat breeding, InternVL2-8B with SFT plus RAG plus RLHF, is reported to beat general chatbots on five breeding tasks and to predict yield with an R-squared of 0.821.","lead":"The authors built a wheat-breeding chatbot by adding domain knowledge to open-source multimodal language models with supervised fine-tuning, retrieval-augmented generation, and reinforcement learning from human feedback. The best version, based on InternVL2-8B, is reported to outperform general-purpose chatbots on breeding questions and to explain 82 percent of yield variation when given drone, weather, plant, and seed data together.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark may leak plot-level field data into SFT training, so WBLM's reported superiority could reflect memorization rather than breeding competence.","rationale":"The reader's weakest assumption targets the validity of the self-constructed evaluation benchmark, and I reach the same conclusion: the reported comparisons cannot be trusted without evidence that the benchmark is independent of the SFT training data. The manuscript itself discloses the key risk in Section 3.1, where 20% of the SFT QA pool is repurposed as the accuracy test for phenotyping and environmental-stress tasks, but it never describes the stratification unit of the split. Since the SFT QA pairs are built from the same field measurements that the benchmark queries, plot-level or date-level overlap is a concrete mechanism for inflating all downstream accuracy, stability, and reasoning comparisons. The paper also releases the dataset, which is a genuine contribution and could enable the proposed test; however, no code or model weights are provided, so the reported yield-prediction numbers in Section 4.2.2 remain unverifiable as written. My concern sharpens the reader's point without changing the verdict: the manuscript should be revised to specify the split at the experimental-unit level, provide external validation or inter-annotator agreement, and give a complete protocol for the yield-prediction experiment. Until then, REJECT remains the appropriate verdict.","tokens_in":25050,"tokens_out":6531,"duration_ms":66685,"concrete_test":"Obtain the released Zenodo dataset and the benchmark QA pairs, then determine whether the 20% benchmark examples share germplasm plots, varieties, or acquisition dates with the 80% SFT training examples. If the split is disjoint at the plot/date level, the leakage concern is resolved. If not, re-run the phenotyping and environmental-stress evaluations using a split that assigns entire plots or entire years to train versus test, and compare WBLM's metrics with the reported values; a substantial drop would confirm that the benchmark scores were inflated by train/evaluation overlap.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise is that WBLM's superiority over other MLLMs on the newly constructed benchmark reflects genuine domain competence, not train/evaluation overlap. Section 3.1 states that 20% of the SFT QA dataset is used for accuracy testing of the phenotyping-estimation and environmental-stress tasks, and Section 3.2 describes a benchmark built and scored by the same team without reporting inter-annotator agreement. The paper does not state whether the 80/20 split is stratified by experimental unit: the same wheat plot, variety, or imaging date may appear in both the SFT training pairs and the benchmark QA pairs. Because the SFT dataset is generated from the same field measurements (yield, SPAD, LAI, CH, etc.) that the benchmark asks the model to predict, a question-level random split can let the model memorize plot-specific values, inflating reported R2 and accuracy. If this occurs, the central comparison across MLLMs, and the ablation conclusions, do not demonstrate domain knowledge. The yield-prediction headline (R2=0.821, RMSE=489.254 kg/ha) is a separate unverifiable claim: Section 4.2.2 gives no model inputs, feature encoding, train/test partition, or cross-validation scheme, so the number cannot be checked from the manuscript as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes building wheat-breeding multimodal large language models (WBLMs) by combining supervised fine-tuning (SFT), retrieval-augmented generation (RAG), and reinforcement learning from human feedback (RLHF) on top of open-source MLLMs such as Qwen-VL, InternVL, and DeepSeek-VL. The authors construct a cross-domain dataset (UAV remote sensing, phenotyping, weather, germplasm, price) and a self-built evaluation benchmark with five task families. They report that the WBLM based on InternVL2-8B outperforms both other WBLMs and closed-source commercial MLLMs, that ablation experiments show the combination of SFT, RAG, and RLHF improves generation quality, and that using all cross-domain data simultaneously yields the best wheat yield prediction with R2=0.821 and RMSE=489.254 kg/ha.","tokens_in":25300,"tokens_out":3621,"duration_ms":32705,"significance":"If the claims were substantiated, the paper would demonstrate a practical integration of heterogeneous agricultural data into a multimodal language model, with a publicly released dataset and reproducible training pipeline. The strengths are the extensive field data collection, the use of standard SFT/RM/PPO formulations, and the decision to release the data on Zenodo. However, the central comparative claims rest entirely on a self-constructed benchmark whose evaluation protocol has serious validity threats, and the headline yield-prediction number is not verifiable from the manuscript. The significance is therefore conditional on fixing these load-bearing evaluation issues.","major_comments":[{"comment":"The accuracy evaluation for the phenotyping-estimation and environmental-stress tasks uses 20% of the SFT question-answer dataset as a test set. Because this SFT dataset is generated from the same plot-level field measurements (yield, SPAD, LAI, CH, CV, WH, PL, WL, FVC) that the benchmark asks the model to predict, a question-level random split does not prevent the same experimental plot, variety, or imaging date from appearing in both training and test pairs. The manuscript does not state whether the split is stratified by experimental unit or whether near-duplicate questions were removed. Without this information, the reported R2 and accuracy values in Section 4.1 could reflect memorization of plot-specific values rather than domain competence, which would invalidate the central comparison across MLLMs and the ablation conclusions.","section":"Section 3.1"},{"comment":"The yield-prediction result (R2=0.821, RMSE=489.254 kg/ha) is presented as the key achievement of cross-domain data fusion, but the manuscript gives no description of how the WBLM produces a numeric yield, what input features are provided in what representation, what train/test split or cross-validation scheme is used, or what objective is optimized. Figure 7 is labelled a cross-validation scatter plot, yet no cross-validation procedure is described. This number cannot be checked or reproduced from the manuscript as written, and it is a central claim in the abstract and conclusion.","section":"Section 4.2.2"},{"comment":"The benchmark relies on manual scoring for accuracy, stability, and reasoning, with no reported number of annotators, no inter-annotator agreement measure, and no external validation panel. Since the same team that built and tuned the models also constructed the benchmark and scored the outputs, the scores for the accuracy, stability, and reasoning comparisons are at risk of subjective bias. The paper should at minimum report a detailed scoring protocol and agreement statistics; without these, the comparative rankings in Section 4.1 are not independently verifiable.","section":"Section 3.2"},{"comment":"The comparison with closed-source commercial MLLMs is confounded by the fact that only the WBLM has access to the authors' knowledge base and fine-tuning data. Tasks such as seed-price query and target-germplasm screening inherently require the private knowledge base, so the poor performance of ChatGPT, Gemini, Qwen, and ERNIE Bot on those tasks is expected and does not demonstrate superior reasoning ability of WBLM. The paper should either restrict comparative claims to tasks that do not require the private knowledge base or provide an additional baseline that receives the same retrieved context through RAG.","section":"Section 4.1"}],"minor_comments":[{"comment":"The text repeatedly uses 'STF' where 'SFT' is intended; please correct this typo throughout the section.","section":"Section 3.1"},{"comment":"The sentence 'This benchmark aims to integrate challenges from various wheat breeding tasks' appears twice in succession.","section":"Section 3.2"},{"comment":"The statement 'ChatGPT shows leading performance among open-source models' is contradictory; ChatGPT is a closed-source model in the paper's list. It should likely read 'among closed-source models.'","section":"Section 4.1"},{"comment":"The mathematical notation contains formatting artifacts (e.g., 𝐸𝐸, 𝜋𝜋𝜃𝜃, 𝐷𝐷𝐾𝐾𝐾𝐾) that should be cleaned up to standard notation.","section":"Equations (1)–(6)"},{"comment":"The term 'cross-validation scatter plot' is used without a description of the cross-validation scheme; please either describe the procedure or rename the figure.","section":"Figure 7 and Section 4.2.2"}],"recommendation":"reject","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a useful dataset paper wrapped in an evaluation that doesn't yet support its headline claims. The field data collection is serious—two years of UAV, phenotyping, weather, and germplasm data on hundreds of wheat plots, plus a 75k QA dataset and external knowledge base, all released on Zenodo. That is a real resource and the main reason to engage.\n\nWhat's new: a systematic comparison of seven open MLLMs with SFT, RAG, and RLHF combinations for wheat breeding, a five-task benchmark, and an ablation showing the full stack helps. The ablation is useful for practitioners, even if each technique is standard.\n\nSoft spots: the accuracy comparisons for phenotyping and stress tasks are compromised by design. Section 3.1 says 20% of the SFT QA pool is used for evaluation, and nothing indicates the split is stratified by plot or date. If a plot appears in both training and test, the model can memorize plot-specific values. That alone undermines the benchmark superiority claims. The yield prediction headline (R2=0.821) is a separate claim without a method: no inputs, feature encoding, train/test partition, or objective. It cannot be checked from the manuscript. Manual scoring without inter-annotator agreement adds noise. No code or weights, so the SFT/RM/PPO pipeline isn't reproducible.\n\nI'd push back on the reader's 'circularity' framing—the closed-source models were not trained on the benchmark, so the comparison isn't circular. The problem is leakage, not circularity.\n\nWho this is for: agricultural AI researchers and breeders wanting a domain assistant. With a revision that adds the yield prediction protocol, plot-stratified splits, inter-annotator agreement, and code/data artifacts, this could be solid. Right now it deserves a serious referee but should not be accepted as is.","headline":"Useful wheat-breeding dataset and systematic MLLM comparison, but benchmark leakage and a missing yield-prediction protocol undercut the headline claims.","tokens_in":25838,"tokens_out":2123,"would_cite":false,"duration_ms":20415,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a multimodal large language model adapted to wheat breeding with SFT, RAG, and RLHF outperforms general-purpose chatbots on breeding tasks, and that fusing remote sensing, phenotyping, weather, and germplasm data…","keywords":["multimodal large language model","wheat breeding","supervised fine-tuning","retrieval-augmented generation","reinforcement learning from human feedback","yield prediction","UAV remote sensing","cross-domain data fusion"],"falsifier":"Evaluate WBLM on an independently built wheat-breeding question set that has no overlap with the SFT, reward-model, or PPO training data, with answers scored by breeders blind to model identity; if WBLM's accuracy and stability advantages over commercial MLLMs disappear, the benchmark-driven claim is falsified. Also test WBLM yield predictions on a new site or season absent from the training split; an $R^2$ well below 0.821 would show the cross-domain fusion result does not generalize.","tokens_in":24859,"feed_emoji":"🌾","tokens_out":10997,"duration_ms":89998,"temperature":0.7,"pith_summary":"This paper tries to establish that an open-source multimodal large language model can become a practical wheat-breeding assistant when domain knowledge is injected through three complementary techniques: supervised fine-tuning (SFT), retrieval-augmented generation (RAG), and reinforcement learning from human feedback (RLHF). The authors build several WBLMs on different open-source vision-language models and report that the InternVL2-8B version leads on a new evaluation benchmark they designed, outperforming commercial chatbots such as GPT-4o, Gemini, Qwen, and ERNIE Bot on breeding-specific tasks. They further report that using remote sensing, phenotyping, weather, and germplasm data together yields the best wheat-yield predictions, with $R^2=0.821$ and RMSE $=489.254$ kg/ha. If these results hold, breeders could get integrated, traceable decision support from a single conversational tool instead of assembling data and code from many specialized sources.","feed_headline":"Fine-tuned open-source AI model tops wheat-breeding benchmark","feed_subtitle":"SFT, RAG, and RLHF injection lifts yield prediction to R² 0.821 with cross-domain field data.","key_machinery":"The load-bearing machinery is the three-stage knowledge-injection pipeline for building WBLM, together with a new evaluation benchmark. SFT uses LoRA fine-tuning on a 75k-question dataset; RAG retrieves from an external knowledge base (germplasm, cultivation, plant protection, seed price) through a vector-database pipeline with BGE-M3 embeddings and reranking; RLHF trains a reward model and optimizes the policy with PPO. The evaluation benchmark defines five wheat-breeding tasks and scores models on accuracy, stability, and reasoning; it is the instrument that carries the paper's comparative performance claims.","core_discovery":"The central claim is that a domain-adapted multimodal large language model, WBLM, built by applying supervised fine-tuning, retrieval-augmented generation, and reinforcement learning from human feedback to an open-source InternVL2-8B backbone, achieves leading performance on a purpose-built wheat-breeding evaluation benchmark spanning phenotyping estimation, environmental stress assessment, target germplasm screening, cultivation technique recommendation, and seed price query. The paper reports that this WBLM outperforms both other open-source WBLMs and closed-source commercial MLLMs (GPT-4o, Gemini, Qwen, ERNIE Bot) on accuracy and stability, and that it gives the best yield prediction when remote sensing, phenotyping, weather, and germplasm data are fused, with $R^2=0.821$ and RMSE $=489.254$ kg/ha. The authors also claim that the three knowledge-injection techniques are complementary: SFT teaches expected answer formats, RAG supplies traceable external facts, and RLHF aligns answers with breeder preferences, so the combined model produces more professional, stable, and less hallucinated answers than the base MLLM alone.","pith_inferences":["My inference: an independent, externally curated benchmark with no overlap with the training data would provide the cleanest test of whether WBLM's edge over commercial chatbots is due to domain knowledge rather than benchmark familiarity.","My inference: the yield-prediction pipeline should be tested on a new site and season to see whether the $R^2=0.821$ achieved on the two experimental stations transfers to unseen environments.","My inference: the same SFT+RAG+RLHF recipe is portable to other crops and to dynamic knowledge bases, and continuous refresh of the price database would turn the seed-price query subtask into a real-time service.","My inference: using the released dataset, one could isolate how much of the gain comes from RAG's traceable facts versus SFT's formatting by testing retrieval-only and fine-tuning-only variants."],"forward_implications":["If WBLM's performance is real, breeders can query one multimodal assistant for phenotyping estimates, stress assessment, germplasm screening, cultivation advice, and seed prices instead of moving between databases, papers, and custom code.","Cross-domain fusion of UAV remote sensing, field phenotyping, weather, and germplasm data becomes the recommended input set for yield prediction, since the paper reports it beats any single source or pair.","The ablation results imply a reusable recipe: SFT, RAG, and RLHF contribute differently, so future domain-adapted agricultural models should combine all three rather than pick one.","The released benchmark and dataset give other teams a common testbed for comparing future breeding MLLMs.","Off-the-shelf closed-source commercial MLLMs would remain weak on specialized breeding questions unless they allow fine-tuning or retrieval augmentation."],"supporting_citations":[{"why":"Supplies the InternVL2-8B backbone that yields the paper's best-performing WBLM and grounds the scale comparison.","marker":"Z. Chen et al., 2024"},{"why":"Provides the Qwen-VL open-source MLLM used as another WBLM base and comparison point.","marker":"Bai et al., 2023"},{"why":"Provides DeepSeek-VL base models used to construct additional WBLM variants.","marker":"DeepSeek-AI et al., 2024"},{"why":"Supplies BGE-M3 embeddings and reranking used in the RAG retrieval pipeline.","marker":"J. Chen et al., 2024"},{"why":"Supplies the PPO algorithm used in the RLHF stage to optimize the policy.","marker":"Schulman et al., 2017"},{"why":"Demonstrates UAV multimodal data fusion for crop yield prediction, the prior result the paper extends.","marker":"Maimaitijiang et al., 2020"},{"why":"Analyzes RAG versus fine-tuning tradeoffs, which the paper cites to interpret its ablation results.","marker":"Balaguer et al., 2024"},{"why":"Provides a general multimodal benchmark where commercial MLLMs score well, used as contrast to the breeding-specific benchmark.","marker":"Yue et al., 2024"}],"fun_headline_variants":["Wheat breeding AI: fine-tuned open model hits R² 0.821","SFT+RAG+RLHF-tuned wheat model leads on yield, R²=0.821","Open-source wheat AI with knowledge injection beats commercial rivals","Cross-domain wheat data fusion in LLM yields R² 0.821","Fine-tuned open-source MLLM wins wheat breeding tasks, yields at R² 0.821"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the paper's self-constructed wheat-breeding evaluation benchmark is a valid, unbiased test, even though the same team built both the models and the benchmark and reserved some test questions from the SFT training pool.","fun_headline_variants_meta":{"raw":{"variants":["Wheat breeding AI: fine-tuned open model hits R² 0.821","SFT+RAG+RLHF-tuned wheat model leads on yield, R²=0.821","Open-source wheat AI with knowledge injection beats commercial rivals","Cross-domain wheat data fusion in LLM yields R² 0.821","Fine-tuned open-source MLLM wins wheat breeding tasks, yields at R² 0.821"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001427,"raw_usage":{"total_tokens":5839,"prompt_tokens":1107,"completion_tokens":4732,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":723,"completion_tokens_details":{"reasoning_tokens":4622}},"tokens_in":723,"tokens_out":4732,"duration_ms":33268,"temperature":1.0,"reasoning_tokens":4622,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:54:46.446872+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate WBLM on an independently built wheat-breeding question set that has no overlap with the SFT, reward-model, or PPO training data, with answers scored by breeders blind to model identity; if WBLM's accuracy and stability advantages over commercial MLLMs disappear, the benchmark-driven claim is falsified. Also test WBLM yield predictions on a new site or season absent from the training split; an $R^2$ well below 0.821 would show the cross-domain fusion result does not generalize.","supporting_citations":[],"review_version":1}