{"id":"090dc9ba-cc13-46b9-9471-b4341ca7c338","arxiv_id":"2608.13344","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"LongEarth-R1, a vision-language model trained with sequence-aware supervision and spatiotemporal GRPO, sets new state-of-the-art results on all 12 tasks of the new LongEarth-Bench for long-horizon Earth observation reasoning.","lead":"LongEarth-Bench, a new benchmark with 120,000 questions over long satellite image sequences, tests whether vision-language models can reason about how places evolve over time. The authors train LongEarth-R1 with step-by-step reasoning and reinforcement learning, and it outperforms prior remote sensing models on all 12 benchmark tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LongEarth-R1 is trained on LongEarth-Bench and then evaluated on the same benchmark with no described train/test split; the reported 'best on all 12 tasks' results may reflect memorization rather than generalization.","rationale":"The reader's weakest assumption is that the rule-based ground-truth annotations are correct and unambiguous. While that is a real concern, the more immediate and load-bearing issue is that the paper never describes a train/test split for LongEarth-Bench, and the method appears to train and evaluate on the same 120k samples. This is a standard and fatal issue for a benchmark claim: if the model is tested on data it was trained on, the reported SOTA numbers are not evidence of generalization. The paper's explicit handling of training splits for the standard benchmarks (AID, UCM, ABCD-CD, etc.) further suggests that the omission for LongEarth-Bench is not an oversight of terminology but a substantive gap. My recommended verdict is UNVERDICTED rather than REJECT because the paper may have an unreported split, but as written the central empirical claim cannot be evaluated. The reader identified label quality, not train/test contamination; these are distinct concerns, so I disagree with their weakest-assumption selection. The concrete test of checking for a held-out split would settle the matter directly.","tokens_in":513,"tokens_out":3715,"duration_ms":78159,"concrete_test":"Inspect the released data/code for explicit train/test split files for LongEarth-Bench. If no split exists, independently retrain LongEarth-R1 on a random 80% subset of LongEarth-Bench (ensuring no sequence overlaps between split portions) and evaluate on the remaining held-out 20%. Compare per-task accuracy with Table 1. If held-out scores drop by a substantial margin (e.g., more than 10 points on any task), the claim of 'best on all 12 tasks' is not supported.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The central claim is that LongEarth-R1 achieves state-of-the-art results on all 12 long-sequence tasks in LongEarth-Bench (abstract, Table 1). The experimental section evaluates LongEarth-R1 on LongEarth-Bench, but never specifies a training/test split for this benchmark. The method section states that Stage 1 supervised fine-tuning uses LongEarth-Bench answer-only and CoT samples, and Stage 2 GRPO uses the same benchmark with temporal and spatial rewards. The 30k structured reasoning subset is also used for training. Thus, the model appears to be trained and evaluated on the same data. This is not a minor omission: a benchmark is only informative if the evaluated model has not seen the test answers. The paper's own standard-benchmark evaluations (Section 'Performance on Standard Remote Sensing Tasks') explicitly say they fine-tune on the corresponding training split, demonstrating that the authors are aware of the need for held-out evaluation but do not report such a split for LongEarth-Bench. Without a held-out split, the headline 'best on all 12 long-sequence tasks' is unsubstantiated; it could be an artifact of training-set memorization. The reader's concern about label correctness is secondary: even if the rule-based labels were perfect, evaluating on training data would invalidate the comparison. This is the most load-bearing concern because it undermines the primary empirical contribution of the paper.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LongEarth-Bench, a large benchmark for long-horizon Earth observation reasoning containing approximately 120k question-answering samples derived from 117k unique images, with an average sequence length of 15.14 frames and a maximum of 30 frames. The benchmark spans 12 tasks across four cognitive dimensions: evolution summarization, spatial reasoning, anomaly identification, and logical prediction, and includes a 30k-sample subset with structured reasoning traces. The paper also presents LongEarth-R1, a vision-language model built on Qwen2.5-VL-7B, trained with supervised fine-tuning using explicit sequence identifiers and structured chain-of-thought supervision, followed by GRPO with format, temporal, and spatial rewards. The authors report that LongEarth-R1 achieves the best results on all 12 long-sequence tasks while remaining competitive on standard remote sensing benchmarks.","tokens_in":13936,"tokens_out":5142,"duration_ms":48916,"significance":"If the results are valid, LongEarth-Bench fills a clear gap in sequence length and cognitive coverage for remote sensing vision-language models, and LongEarth-R1 demonstrates substantial gains on anomaly identification and long-range temporal reasoning. The release of the benchmark and the two-stage training recipe would be a useful contribution to the community. The ablations support the individual contributions of sequence identifiers, CoT supervision, and the proposed rewards. However, the lack of a described train/test split for LongEarth-Bench is a critical flaw that casts the headline results as potentially reflecting memorization; the paper must address this before the claims can be accepted.","major_comments":[{"comment":"No train/test split is described for LongEarth-Bench. The method section states that Stage 1 SFT and Stage 2 GRPO train on LongEarth-Bench samples, and the experiments evaluate the same benchmark without any held-out split. In contrast, the standard benchmark evaluations (§5, 'Performance on Standard Remote Sensing Tasks') explicitly fine-tune on the corresponding training split, showing that the authors recognize the need for held-out evaluation. Without a disjoint test split, the claimed 'best results on all 12 long-sequence tasks' is unsubstantiated; it could be an artifact of training-set memorization. Please specify and use a held-out test split for LongEarth-Bench and report results on it, or explicitly justify why the current evaluation is not affected by leakage.","section":"§5, Table 1 and §4"},{"comment":"The ground-truth answers are generated by rule-based pipelines and machine-generated CoT traces with only 'rule-based checks', 'human review', and 'spot-check validation'. No quantitative label-quality evidence (e.g., inter-annotator agreement, error rates, number of samples reviewed, or ambiguity resolution) is provided. Since LongEarth-Bench is the sole basis for the claimed state-of-the-art results, label correctness is load-bearing. Please provide a label-quality study with sample sizes, disagreement statistics, and a description of how ambiguous cases were resolved.","section":"§3, Dataset Construction"},{"comment":"The controlled perturbations used to construct anomaly tasks (order perturbations, repeated-frame insertions, contextual disturbances) may create non-visual shortcuts. For example, redundancy detection (T8) could be solved by exact pixel matching of repeated frames without any long-horizon reasoning. The paper should demonstrate that models cannot exploit such low-level cues, for instance by adding noise or randomizing frame order at evaluation time, and should report human performance to calibrate the difficulty and validity of these tasks.","section":"§3, Anomaly tasks (T7–T9)"}],"minor_comments":[{"comment":"The figures are extremely dense, with small fonts and many overlaid elements; please enlarge and simplify them for readability.","section":"Figure 1 and Figure 2"},{"comment":"The reward definitions are said to be in the supplementary material, but no supplementary file is provided; please include the detailed reward formulas and hyperparameters (λ_f, λ_t, λ_s, G, β, ε) in an appendix.","section":"§4, Spatiotemporal Alignment via GRPO"},{"comment":"The GRPO hyperparameters (group size G, KL coefficient β, clipping coefficient ε) and the reward weights are not reported in the main text; please provide them for reproducibility.","section":"§5, Experimental Setups"},{"comment":"The row headers in the upper block of the ablation table are ambiguous; consider using a cumulative notation (e.g., '+' for adding each component) to clarify that each row includes all previous components.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"If the authors cannot provide a held-out test split for LongEarth-Bench, I would recommend rejection; the current evaluation is potentially circular. Otherwise, the benchmark itself is a useful contribution and the paper should be accepted after major revision. The reader's report's concern about label correctness is secondary, but the dataset quality evidence should be strengthened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: LongEarth-Bench is a genuinely new and useful asset, but the paper's headline numbers are currently not interpretable because the model is trained on the very benchmark it is then evaluated on, with no train/test split described anywhere. That is the elephant in the room.\n\nWhat is actually new: the benchmark itself. 120k QA samples, average 15.14 frames (max 30), 12 tasks across evolution summarization, spatial reasoning, anomaly identification, and logical prediction, built from five public datasets. That is a real step up from TEOChat's 2-frame average. The 30k structured reasoning subset is a nice addition, even if the traces are machine-generated by Qwen3-VL-8B-Thinking with only spot-check human validation. The training recipe—SFT with explicit sequence IDs plus GRPO with format/temporal/spatial rewards—is sensible and well-motivated, and the ablations are thorough. Table 2 also shows the model transfers reasonably to standard RS benchmarks when fine-tuned on their training splits.\n\nThe soft spots, in order of severity. First and most important: there is no train/test split for LongEarth-Bench. The method section says Stage 1 SFT uses LongEarth-Bench samples and Stage 2 GRPO uses the same benchmark; the experiments report accuracy on LongEarth-Bench with no mention of held-out sequences. The baselines marked with * are fine-tuned on LongEarth-Bench too, so everyone is being evaluated on training data. This makes the 'best on all 12 tasks' claim unverifiable. It could be memorization. The authors know how to do proper held-out evaluation—they diligently fine-tune on AID, UCM, etc. training splits—so the omission in the core evaluation looks like an oversight, not a style choice. Second: no error bars or significance tests anywhere. The gaps between LongEarth-R1 and the best baseline on some tasks (e.g., T11: 89.87 vs 87.62) might be noise. Third: the CoT traces are machine-generated; the paper reports automatic verification and human review but no numbers on inter-rater agreement or how often traces were rejected. Fourth: no release plan for data or code is stated.\n\nThe benchmark construction itself is careful: diverse sources, rule-based answer generation with human review, and controlled perturbations for anomaly tasks. If the authors release the data with a proper split and show long-horizon gains on held-out sequences, this becomes a solid contribution. As it stands, the central claim is unsupported.\n\nWho is this for: anyone working on RSVLMs or long-sequence multimodal reasoning. It is worth reading as an example of benchmark design, and as a cautionary tale about evaluation.\n\nMy recommendation: send it to peer review, but the reviewers should demand a held-out split, error bars, artifact release, and more transparent CoT validation before the numbers are taken seriously.","headline":"Useful benchmark, but the headline results are unverifiable because the model is trained and evaluated on the same data without a held-out split.","tokens_in":14414,"tokens_out":3530,"would_cite":false,"duration_ms":31187,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a vision-language model trained on long satellite image sequences with temporal and spatial rewards—LongEarth-R1—achieves the best results on all 12 long-horizon Earth observation reasoning tasks while staying…","keywords":["long-horizon reasoning","Earth observation","vision-language models","remote sensing benchmark","chain-of-thought","group relative policy optimization","temporal grounding","spatial reasoning"],"falsifier":"Re-run the evaluation on a human-verified held-out subset of sequences whose labels are independently re-derived from the raw segmentation, polygon, land-cover, and flood evidence, and check whether LongEarth-R1 still leads on all 12 tasks; if a nontrivial fraction of the original labels contain wrong frame ranges, mislocated anomalies, or inconsistent spatial extents, the reported gains could be partly an artifact of the rule-based annotation pipeline.","tokens_in":1763,"feed_emoji":"🛰️","tokens_out":2021,"duration_ms":59336,"temperature":0.7,"pith_summary":"Long-horizon Earth observation reasoning means making sense of how a place changes across many satellite images over time, including stages of evolution, where and when changes happen, anomalies in the sequence, and what comes next. The paper argues that existing remote sensing vision-language models mostly handle single images, image pairs, or short clips, so they cannot reliably ground conclusions in the right frames and regions. It introduces LongEarth-Bench, about 120k question-answer samples built from 117k images, with sequences averaging 15.14 frames and reaching 30 frames, covering 12 tasks across four cognitive dimensions. It then trains LongEarth through supervised fine-tuning with explicit sequence identifiers and structured reasoning traces, and LongEarth-R1 through reinforcement learning with format, temporal, and spatial rewards. The central claim is that LongEarth-R1 achieves the best result on every one of the 12 long-sequence tasks while remaining competitive on standard remote sensing benchmarks.","feed_headline":"Model tops all 12 long-horizon Earth observation tasks","feed_subtitle":"New benchmark and temporal-spatial rewards lift vision-language reasoning across 30-frame satellite sequences.","key_machinery":"The load-bearing mechanism is the combination of explicit sequence identifiers and spatiotemporal rewards. Each frame is labeled Image 1 through Image T in the multimodal input, giving the model stable temporal anchors; a 30k-sample structured reasoning subset supplies chain-of-thought traces that connect key frames, changed regions, and final answers. On top of supervised fine-tuning, GRPO optimizes a weighted reward $R_i = \\lambda_f R^{\\mathrm{fmt}}_i + \\lambda_t R^{\\mathrm{time}}_i + \\lambda_s R^{\\mathrm{space}}_i$ that checks answer format, temporal ordering and key-frame matching, and spatial agreement on regions, directions, and extents. This machinery is what shifts the model from imitating answers to producing temporally ordered, spatially grounded reasoning.","core_discovery":"LongEarth-R1 establishes, on the paper's own evaluation, that long-horizon spatiotemporal reasoning in remote sensing can be trained and measured. On LongEarth-Bench it outperforms all compared baselines on all 12 tasks, with particularly strong gains on anomaly identification and tasks requiring long-range temporal evidence. LongEarth-R1 also transfers to conventional benchmarks, achieving the best result on six of nine single-image, bi-temporal, and short-sequence datasets, including all bi-temporal change-understanding benchmarks. The discovery is two-part: a benchmark that operationalizes long-horizon reasoning through four cognitive dimensions, and a training recipe—explicit sequence identifiers plus structured chain-of-thought, then group relative policy optimization with format, temporal, and spatial rewards—that turns a general vision-language model into a state-of-the-art long-sequence Earth observation reasoner.","pith_inferences":["Editorial inference: because the model is trained and evaluated on the same benchmark-derived protocol, the headline result is best read as proof of capability on LongEarth-Bench rather than proof of general long-horizon reasoning; a held-out set of unseen Earth observation sequences would test transfer.","Editorial inference: the same decomposition into format, temporal, and spatial rewards could be carried over to other sequence reasoning domains such as video question answering, climate trajectory prediction, or multi-temporal medical imaging, where frame-index grounding is similarly missing.","Editorial inference: if the rule-based labels contain systematic errors in frame ranges or anomaly placements, the temporal and spatial rewards would reinforce those errors; an independent label audit on a random subset would quantify that risk.","Testable extension: varying the number of input frames and the question's evidence span independently would clarify whether LongEarth-R1's robustness comes primarily from sequence grounding, from reward alignment, or from the base model's scale."],"forward_implications":["Long-horizon Earth observation reasoning can be benchmarked and trained end-to-end, with sequences up to 30 frames tractable for current vision-language models when explicit temporal anchors are provided.","Gains concentrate on anomaly identification tasks such as chronological violations, redundancy detection, and contextual robustness, indicating that temporal-order supervision is what unlocks these abilities.","Long-horizon training transfers to conventional benchmarks, with the best results on all bi-temporal change-understanding datasets, so the long-sequence specialization does not sacrifice general remote sensing understanding.","All tested methods degrade as input sequence length grows, but LongEarth-R1 degrades least, retaining 51.4 accuracy on 26-30-frame inputs after reaching 87.8 on 2-5-frame inputs, showing long-context robustness remains the open frontier.","A wider evidence span generally helps LongEarth-R1 when complementary observations support the answer, with one exception at 23-30 frames, so cross-frame evidence integration is a learned skill rather than a simple length penalty."],"supporting_citations":[{"why":"Base model Qwen2.5-VL-7B that LongEarth and LongEarth-R1 are built from; supplies the frozen visual encoder and trainable language backbone.","marker":"Bai et al. 2025b"},{"why":"SpaceNet 7, source of urban construction sequences with building polygons used to construct several benchmark tasks.","marker":"Van Etten et al. 2021"},{"why":"SDSU MidWest Flood, source of flood dynamics sequences covering disaster evolution and extent assessment.","marker":"Jang et al. 2024"},{"why":"DynamicEarthNet, source of daily multi-spectral natural land-cover evolution sequences.","marker":"Toker et al. 2022"},{"why":"FLAIR#2, source of cloud-snow interference and semantic segmentation sequences.","marker":"Garioud et al. 2023"},{"why":"PASTIS-R, source of crop phenology sequences used for logical prediction and missing-frame tasks.","marker":"Sainte Fare Garnot, Landrieu, and Chehata 2022"},{"why":"TEOChat, the main temporal Earth observation assistant baseline and the protocol used for standard benchmark evaluation.","marker":"Irvin et al. 2025"},{"why":"Qwen3-VL-Thinking used to generate the 30k structured reasoning traces in the benchmark.","marker":"Bai et al. 2025a"},{"why":"Introduces GRPO, the reinforcement-learning algorithm used in Stage 2 for reward-based policy optimization.","marker":"Shao et al. 2024"}],"fun_headline_variants":["LongEarth-R1: top scores on all 12 long-horizon tasks","Benchmark plus RL rewards: VLMs master 30-frame satellite sequences","Earth observation reasoning: model wins all 12 long-sequence tasks","New benchmark and rewards push satellite reasoning to 30 frames"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"The benchmark's ground-truth answers and temporal and spatial labels, generated by rule-based pipelines from segmentation, polygon, land-cover, and flood evidence, are correct and unambiguous across all 12 tasks, so scores measured on it faithfully reflect long-horizon reasoning ability.","fun_headline_variants_meta":{"raw":{"variants":["LongEarth-R1: top scores on all 12 long-horizon tasks","Benchmark plus RL rewards: VLMs master 30-frame satellite sequences","Earth observation reasoning: model wins all 12 long-sequence tasks","New benchmark and rewards push satellite reasoning to 30 frames"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001137,"raw_usage":{"total_tokens":4700,"prompt_tokens":901,"completion_tokens":3799,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":3723}},"tokens_in":517,"tokens_out":3799,"duration_ms":27627,"temperature":1.0,"reasoning_tokens":3723,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:15:02.941483+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the evaluation on a human-verified held-out subset of sequences whose labels are independently re-derived from the raw segmentation, polygon, land-cover, and flood evidence, and check whether LongEarth-R1 still leads on all 12 tasks; if a nontrivial fraction of the original labels contain wrong frame ranges, mislocated anomalies, or inconsistent spatial extents, the reported gains could be partly an artifact of the rule-based annotation pipeline.","supporting_citations":[],"review_version":1}