{"id":"b5e01d0e-31b2-45c1-8e78-8106fbe098d9","arxiv_id":"2506.10228","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A multi-modal transformer trained on Landsat, climate, ET, and soil data reaches R²=0.76 for county-level yield forecasts of 70+ California crops on held-out test years 2021-2022.","lead":"This paper introduces a public benchmark dataset that combines satellite, weather, evapotranspiration, and soil data to forecast county-level yields for more than 70 crops in California, reporting an R² of 0.76 on held-out years. A generalist reader might care because a standardized multi-crop dataset could make agricultural forecasting models easier to compare, reuse, and test across different regions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pooled test R2 of 0.76 may be driven by crop-identity fixed effects rather than year-to-year forecasting skill; without per-crop R2 or trivial baselines the central performance claim is not yet substantiated.","rationale":"The reader set CONDITIONAL based on missing baselines, missing artifacts, and CDL alignment. My stress-test sharpens the condition: the central claim is an aggregate R2 that is not evidence of forecasting unless decomposed. This is more load-bearing than CDL alignment because even a perfect CDL mask would not fix the interpretability problem. The paper's own architecture makes the problem salient, since crop identity, static soil, and county-level aggregation can encode fixed effects. The proposed test is cheap and decisive: a historical-mean baseline is trivial to compute. I agree with the reader's overall conditional verdict; the condition should explicitly require reporting per-crop and within-crop temporal R2 plus baselines. The critique is on the metric choice and missing controls, not on the authors' integrity.","tokens_in":9813,"tokens_out":4596,"duration_ms":55418,"concrete_test":"Use the released code to compute, on the same 2021-2022 test split: (a) pooled R2 as reported; (b) baseline R2 predicting each test sample by the crop-county mean over 2008-2018 (optionally plus a linear trend); (c) within-crop-county R2, computed by subtracting the crop-county mean from both observed and predicted yields before calculating R2 (equivalently, residualize both on crop-county fixed effects and recompute R2). If baseline (b) yields pooled R2 near 0.76, or if (c) is near zero or negative, the claim of strong forecasting performance is unsupported, and the paper should be revised to emphasize cross-crop/cross-county variation rather than temporal forecasting.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper reports one aggregate R2=0.76 on 2021-2022 pooled across more than 70 crops. Because yield levels differ by orders of magnitude across crop types (Figure 2) and crop identity is explicitly provided as a categorical input (Section 3), a model that simply predicts the historical crop-county mean yield for each test sample would obtain a high pooled R2 without forecasting any interannual variation. Table 2 shows test RMSE values (e.g., corn 10.10 ton/ac, almond 8.41 ton/ac) that are large relative to typical yields, suggesting weak within-crop temporal prediction, but no per-crop R2 or within-crop temporal R2 is reported. No baselines are given, not even a trivial predictor using the training-period crop-county mean, a linear trend, or lagged yield. Therefore the headline number conflates between-crop and between-county variance with genuine forecasting skill. The stated objective is accurate and timely crop yield forecasting; as reported, the evidence does not establish that the model forecasts year-to-year deviations. The CDL-mask alignment issue raised by the reader is real but secondary: misalignment would add noise and attenuate signal, whereas the pooled-R2 issue determines whether the central number is interpretable at all.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces a county-level crop yield benchmark dataset covering over 70 crops in California from 2008 to 2022 (excluding 2012), integrating Landsat satellite imagery, Daymet climate records, OpenET evapotranspiration, and SSURGO soil properties, aligned to USDA NASS county-level yield reports via CDL-derived crop field masks. The authors also propose a multi-modal Vision Transformer architecture that combines these data layers with crop identity as a categorical input, and they report an overall R² of 0.76 on a held-out test set comprising 2021–2022.","tokens_in":10034,"tokens_out":3785,"duration_ms":43663,"significance":"The dataset itself, if released with its documented preprocessing pipeline, would be a valuable community resource: it spans an unusually broad set of crops, covers all California counties, and integrates four complementary data modalities at high spatial resolution. The proposed architecture is a reasonable extension of existing multi-modal ViTs. However, the central performance claim is not substantiated by the evidence presented. The pooled R² conflates between-crop yield-level differences with year-to-year forecasting skill, and the paper provides no baselines, no per-crop or within-crop R², no uncertainty estimates, and no ablations. The current evidence supports the dataset contribution more strongly than the forecasting-performance claim, and the paper needs substantial additional analysis before the model's predictive ability can be assessed.","major_comments":[{"comment":"The headline R² of 0.76 is pooled across all crops, and crop identity is explicitly provided as a categorical input (Section 3, Figure 6). Because yield levels differ by orders of magnitude across crops (Figure 2), a model that simply predicts the training-period crop-county mean yield for each test sample would already achieve a high pooled R² without forecasting any interannual variation. The paper does not report per-crop R², within-crop temporal R², or any baseline such as a crop-county historical mean, a linear trend, or a lagged-yield predictor. Without such comparisons, the central claim of 'strong predictive performance' is not supported. Please report per-crop and within-crop R² and compare against at least the trivial crop-county mean baseline.","section":"Section 3, Figure 6, and Results (Section 4)"},{"comment":"The per-crop test RMSE values are presented without reference to the yield scale or to baseline errors. For example, corn test RMSE is 10.10 ton/ac and almond is 8.41 ton/ac, which appear large relative to typical yields of those crops, yet no per-crop baseline error is given. Additionally, the units are inconsistent with Figure 2 and the abstract's mention of ton/ha. Please report relative errors (e.g., RMSE divided by mean yield) and compare against a per-crop historical mean baseline for the same test years.","section":"Table 2"},{"comment":"The paper reports results from a single temporal split (train 2008–2018, validation 2019–2020, test 2021–2022) with no standard deviation or error bars, despite mentioning 5-fold cross-validation. It is therefore impossible to assess whether the difference between validation R² of 0.667 and test R² of 0.767 is meaningful or within noise. Please provide mean and standard deviation across cross-validation folds or random seeds for all reported metrics.","section":"Section 3.1"},{"comment":"The contribution is framed around integrating four data modalities, but no ablation or comparison against single-modality variants is provided. The paper therefore does not demonstrate the marginal value of evapotranspiration, soil, climate, or satellite data. Please include ablations (e.g., removing each modality in turn) or at least a comparison against a model that uses only a subset of the modalities.","section":"Sections 2–3"},{"comment":"The model assumes that CDL-derived field masks (fields at least 10 hectares) and the corresponding pixel-level features are aligned with the fields that contribute to the USDA county-level yield statistics. If CDL crop boundaries, rotations, or planting dates do not match the fields captured in the county yield reports, the learned mapping from pixels to yield will be attenuated or biased. This potential misalignment is not discussed as a limitation; please add a discussion of its likely impact and, if possible, sensitivity analyses using alternative CDL-based masking thresholds.","section":"Section 2.3"}],"minor_comments":[{"comment":"The paper alternates between 'ton/ac' and 'ton/ha' (Table 2 vs. Figure 2) without stating the conversion or the intended unit. Please unify units and specify them consistently in all figures, tables, and text.","section":"Throughout"},{"comment":"The section title 'Multi-Model Vision Transformers' should be 'Multi-Modal Vision Transformers' to match the architecture described.","section":"Title and Section 3"},{"comment":"The architecture description is vague: the stratified sampling strategy, the handling of mismatched pixel counts across modalities (N vs. M), and the mechanism for fusing a 365-step daily climate sequence with a 12-step monthly sequence are not specified. Please clarify these details, as they are essential for reproducibility.","section":"Section 3.1 and Figure 6"},{"comment":"The evaluation metrics R², RMSE, and MAE are not formally defined. Please provide their definitions and specify whether R² is computed on the pooled set of all crop-year-county observations.","section":"Section 4"},{"comment":"The paper states 'over 70 crops' and '70 different crops' inconsistently; please report the exact number of crops included in the benchmark.","section":"Abstract and Section 1"}],"recommendation":"major_revision","confidential_remarks":"I would encourage the editor to treat this manuscript primarily as a dataset/benchmark contribution. The dataset is potentially valuable and the code/dataset release is commendable, but the model's reported performance is not a reliable selling point without baselines and per-crop analyses. The authors should either substantially tone down the model claims or add the missing experiments. The reader's circularity concern is not well-founded in my view: labels come from independent USDA reports, test years are held out, and crop identity is legitimate input information; the real issue is that the pooled R² can be inflated by between-crop variance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the benchmark dataset is the real contribution here; the modeling results, as reported, do not demonstrate the forecasting skill they claim. The pooled test R2 of 0.76 across 70 crops is nearly uninterpretable without a per-crop breakdown, because crop type is an input and crop yields differ by orders of magnitude. A model that just predicts the mean yield for each crop-county would get a high pooled R2 while having zero year-to-year skill. Table 2's RMSEs for corn, almonds, and walnuts are large relative to typical yields, which supports the concern. The paper needs baselines (e.g., long-run crop-county mean, lagged yield, linear trend), per-crop R2, and ideally a within-crop temporal R2, plus error bars over folds or seeds.\n\nOn the positive side, assembling a consistent multi-modal dataset for 70+ crops across all California counties from Landsat, Daymet, OpenET, and SSURGO is genuinely useful. That alone is a reason to engage. The data processing decisions—monthly Landsat composites, CDL-based field masks, static soil rasterization—are clearly described. The manuscript is transparent about the temporal split and the missing field-level ground truth.\n\nThe soft spots beyond the pooled metric: there are no baselines or ablations at all, no repository URL or commit hash despite the claim that the code is public, and the CDL-to-county-yield alignment is worth checking—USDA county statistics include all fields, not just the >10 ha fields they extract. That mismatch could attenuate the learned relationships. These are fixable, but they need to be fixed before the R2 claim means anything.\n\nWho benefits: anyone wanting a ready-made multi-crop, multi-modal benchmark for California county-level yield forecasting. A serious referee should be given this paper, because the dataset contribution is real and the claims are testable once the artifacts are released. My own verdict is skeptical, but the right response is peer review with a demand for baselines and stricter metrics, not a desk reject.","headline":"A genuinely useful California crop benchmark, but the pooled R2 claim as reported is not yet substantiated because it likely reflects crop identity rather than year-to-year forecasting skill.","tokens_in":10613,"tokens_out":2047,"would_cite":false,"duration_ms":24087,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single multi-modal network, fed satellite, climate, evapotranspiration, and soil data aligned to crop fields, forecasts county-level yields for over 70 California crops at $R^2$ 0.76 on unseen test years.","keywords":["crop yield forecasting","multi-modal deep learning","Vision Transformer","county-level benchmark dataset","Landsat satellite imagery","California agriculture","evapotranspiration","soil properties"],"falsifier":"Compute per-crop $R^2$ on the 2021–2022 test set: if most of the 70 crops sit near or below zero individually while the pooled $R^2$ is 0.76, the headline is an artifact of pooling crops with different yield scales rather than evidence of across-crop generalization. A second check compares prediction errors on county–crop pairs where mapped crop acreage closely matches the reported harvested acreage against pairs where the two diverge sharply; if errors do not grow with that divergence, the field-mask alignment assumption is not doing the load-bearing work.","tokens_in":9574,"feed_emoji":"🌾","tokens_out":16522,"duration_ms":162107,"temperature":0.7,"pith_summary":"The paper claims that one multi-modal deep network, trained on Landsat satellite imagery, daily Daymet climate records, monthly OpenET evapotranspiration, and high-resolution SSURGO soil maps covering every county in California from 2008 to 2022 (except 2012), can forecast county-level yields for more than 70 crops. Crop identity is fed to the model as an input, pixels are sampled inside crop-field masks, and the held-out test years 2021–2022 yield an overall $R^2$ of 0.76. If correct, yield forecasting no longer has to be rebuilt crop by crop, since a single trained model spans grains, fruits, vegetables, forage, and industrial crops across the state. The authors release the dataset and code, so the central claim can be checked and extended directly.","feed_headline":"One model forecasts 70+ California crops at R² 0.76","feed_subtitle":"A single network spanning over 70 crops predicts held-out county yields from satellite, weather, water, and soil data.","key_machinery":"The central object is a multi-modal Vision Transformer with a stratified feature-extraction pipeline: because each crop field contains different numbers of pixels and the data sources arrive at different resolutions (30 m for imagery, ET, and soil; 1 km for climate), each modality is subsampled by a stratified strategy so that every input becomes a fixed-size embedding of dimension 256. Monthly Landsat ($12\\times6$), monthly ET ($12\\times1$), daily climate ($365\\times8$), static soil ($1\\times5$), and a categorical crop identity ($1\\times1$) are encoded separately, then fused by an 8-layer, 6-head transformer encoder whose time-series regression head outputs the county-level yield estimate. Feeding crop identity as a constant input is what lets one model span 70 crops, and injecting static soil at every time step is how the model keeps long-term soil influence explicit throughout the growing season.","core_discovery":"The paper's central claim is that pixel-level samples of six Landsat reflectance bands (monthly), eight Daymet climate variables (daily), OpenET evapotranspiration (monthly), and five SSURGO soil attributes (static), taken inside crop-field masks of at least 10 hectares, carry enough signal for a single multi-modal Vision Transformer to predict county-level yield across 70 crops with an overall $R^2$ of 0.76 on the 2021–2022 test years. The authors report a training $R^2$ of 0.864 and a validation $R^2$ of 0.667, and their per-crop table shows that grapes, greens, and tomatoes predict tightly while corn, almonds, and walnuts carry test RMSE above 8 ton/ac, which they attribute to higher variability and sparser samples. The stated contribution is a reproducible, public benchmark and framework that moves multi-modal yield forecasting from one or two major crops to the full diversity of a state's agriculture.","pith_inferences":["A per-crop $R^2$ table for the held-out years would show how much of the pooled 0.76 comes from crops with many county–year samples; the paper's own table shows grapes and greens testing well while corn, almonds, and walnuts test with RMSE above 8 ton/ac, so the headline masks a wide spread.","No ablation is reported, so the claim that all four data layers (imagery, climate, ET, soil) are needed is untested; dropping each modality in turn would reveal which layers actually carry the signal and which add little.","Because similar satellite, weather, evapotranspiration, crop-mask, and soil products cover other states, retraining the identical pipeline outside California is a direct test of whether the learned relationships are general or state-specific.","The validation $R^2$ (0.667) sitting below the test $R^2$ (0.767) is an unusual ordering; checking whether 2021–2022 are calmer climate years or whether per-county composition differs would clarify what the headline number means."],"forward_implications":["If the $R^2$ of 0.76 holds, one trained network can replace per-crop forecast models for the benchmark's 70 crops, with the crop name supplied as an input.","The released dataset standardizes county-level inputs (monthly Landsat, daily climate, monthly ET, static soil) for 2008–2022 except 2012, letting future models be compared on identical data.","The crops that test poorly, which the paper lists as corn, almonds, and walnuts with RMSE above 8 ton/ac, mark where current inputs are insufficient and where the authors say longer time series and crop-specific features are needed.","The authors state the framework can be retrained at field level if field-level yield measurements become available, so the same code path serves both county-level and field-level forecasting."],"supporting_citations":[{"why":"County-level crop yield reports serving as the ground-truth labels the model is trained and tested against.","marker":"[25]"},{"why":"Annual 30 m crop-type maps that define cultivated field boundaries (at least 10 hectares) for pixel extraction.","marker":"[24]"},{"why":"Daily 1 km gridded weather supplying the eight climate variables used as the climate modality.","marker":"[22]"},{"why":"Monthly 30 m evapotranspiration estimates from the OpenET ensemble, the water-balance modality.","marker":"[18]"},{"why":"Soil survey attributes for the five static soil properties rasterized to 30 m.","marker":"[20]"},{"why":"Prior multi-modal fusion model for vineyard yield whose architecture this benchmark extends to 70 crops.","marker":"[12]"},{"why":"Multi-modal spatial-temporal transformer for major-crop yield prediction whose scope this work broadens.","marker":"[16]"},{"why":"The attention mechanism that the multi-modal transformer encoder is built on.","marker":"[26]"}],"fun_headline_variants":["Multi-modal model forecasts 70+ crops at R² 0.76","One neural net predicts yields for 70+ California crops","Satellite+climate+soil data forecast 70 crops with R² 0.76","A single ViT predicts 70 crops across all CA counties","Benchmark: 70 crops, 4 data types, R² 0.76 on test set"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method assumes that the crop-field maps used to choose which pixels to read match the actual fields whose harvests the county yield statistics summarize; if the maps mark the wrong fields, miss crop rotations, or use different planting seasons, the learned link from imagery to yield is weakened or shifted.","fun_headline_variants_meta":{"raw":{"variants":["Multi-modal model forecasts 70+ crops at R² 0.76","One neural net predicts yields for 70+ California crops","Satellite+climate+soil data forecast 70 crops with R² 0.76","A single ViT predicts 70 crops across all CA counties","Benchmark: 70 crops, 4 data types, R² 0.76 on test set"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000489,"raw_usage":{"total_tokens":2436,"prompt_tokens":1005,"completion_tokens":1431,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":621,"completion_tokens_details":{"reasoning_tokens":1328}},"tokens_in":621,"tokens_out":1431,"duration_ms":12643,"temperature":1.0,"reasoning_tokens":1328,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:31:17.679918+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute per-crop $R^2$ on the 2021–2022 test set: if most of the 70 crops sit near or below zero individually while the pooled $R^2$ is 0.76, the headline is an artifact of pooling crops with different yield scales rather than evidence of across-crop generalization. A second check compares prediction errors on county–crop pairs where mapped crop acreage closely matches the reported harvested acreage against pairs where the two diverge sharply; if errors do not grow with that divergence, the field-mask alignment assumption is not doing the load-bearing work.","supporting_citations":[{"cited_title":"USDA Na- tional Agricultural Statistics Service, 2020","cited_arxiv_id":null,"evidence_quote":"County-level crop yield reports serving as the ground-truth labels the model is trained and tested against."},{"cited_title":"Department of Agriculture (USDA)","cited_arxiv_id":null,"evidence_quote":"Annual 30 m crop-type maps that define cultivated field boundaries (at least 10 hectares) for pixel extraction."},{"cited_title":"Daymet: Daily surface weather data on a 1-km grid for north america, version 2","cited_arxiv_id":null,"evidence_quote":"Daily 1 km gridded weather supplying the eight climate variables used as the climate modality."},{"cited_title":"Openet: Filling a critical data gap in water management for the western united states.JAWRA Journal of the American Water Resources Association, 58(6):971–994, 2022","cited_arxiv_id":null,"evidence_quote":"Monthly 30 m evapotranspiration estimates from the OpenET ensemble, the water-balance modality."},{"cited_title":"Department of Agriculture","cited_arxiv_id":null,"evidence_quote":"Soil survey attributes for the five static soil properties rasterized to 30 m."},{"cited_title":"CMAViT: Integrating Climate, Managment, and Remote Sensing Data for Crop Yield Estimation with Multimodel Vision Transformers","cited_arxiv_id":"2411.16989","evidence_quote":"Prior multi-modal fusion model for vineyard yield whose architecture this benchmark extends to 70 crops."},{"cited_title":"Mmst-vit: Climate change- aware crop yield prediction via multi-modal spatial-temporal vision transformer","cited_arxiv_id":null,"evidence_quote":"Multi-modal spatial-temporal transformer for major-crop yield prediction whose scope this work broadens."}],"review_version":1}