{"id":"e2403329-d156-48cf-b72f-87d7d8f6a568","arxiv_id":"2608.10494","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":10,"one_line_summary":"GeoForge improves Earth-observation LLM agents by distilling completed tool trajectories into three retrievable memory components that guide future planning and tool use.","lead":"GeoForge is an agent framework that stores successful Earth observation tool-use trajectories as a structured, retrievable memory and reuses them to guide new tasks without retraining the language model. The authors report accuracy gains over baseline agents on several geospatial benchmarks, but the paper does not yet release code or state whether the memory was evolved on the same tasks used for evaluation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Memory-evolution/evaluation split is unspecified, so the reported accuracy gains may reflect test-set leakage into retrieved priors.","rationale":"The reader's weakest assumption—that memory-updated trajectories are disjoint from evaluated queries—is precisely the most load-bearing point. The core mechanism of GeoForge is retrieval of task-conditioned priors from memories that evolve over time; if those priors encode answer-relevant information from the evaluation set, the central claim of consistent improvement is unverifiable. The paper provides no explicit statement of the split in the Evaluation Protocols or the Safety-Gated Self-Evolution sections, and the phrase 'After each task, a safety-gated distillation process...' suggests online evolution during evaluation, making the protocol ambiguous. This concern is independent of the paper's internal consistency; it directly tests whether the method does what is claimed. The reader's CONDITIONAL verdict remains appropriate because the issue is addressable with a disclosed protocol and re-runs. I also note an additional inconsistency supporting the concern about overclaiming: Table 1 shows GPT-4o Tool-In-Order drops from 53.20 (Earth-Agent) and 48.93 (GeoEvolver) to 37.31, contradicting the abstract's claim of consistent trajectory-quality improvements. This further justifies requiring a careful protocol disclosure. The proposed concrete test—re-running with a disjoint memory construction set—would settle whether the headline improvements are real or a leakage artifact.","tokens_in":12350,"tokens_out":4307,"duration_ms":42445,"concrete_test":"Require the authors to specify the exact evolution/evaluation split for each benchmark and re-run Earth-Bench with a clearly disjoint protocol: build memory from a random 50% subset of tasks, freeze retrieval, and evaluate on the remaining 50% with the same backbones and baselines. If the accuracy gap over GeoEvolver shrinks materially or reverses, the reported self-evolution gains are contaminated by test-set leakage rather than generalizable execution knowledge.","verdict_should_be":"UNCHANGED","load_bearing_attack":"GeoForge's central claim is that retrieving from self-evolving memories improves accuracy and trajectory quality. The load-bearing assumption is that memory updates (Eq. 18, via the safety-gated distillation in 'Safety-Gated Self-Evolution') are fed by trajectories disjoint from the evaluation queries. The 'Evaluation Protocols' paragraph states only that official benchmark protocols are followed; it never specifies whether memory is initialized empty at evaluation, pre-populated from the same benchmark tasks, or grown online across the test set. If memory is warm-started from prior runs on the same tasks, retrieved Workflow Graph nodes, Action-Level Experiences, and Adapted SOPs can contain the exact tool sequences, parameter values, and cautionary notes needed to answer those queries, inflating both accuracy and trajectory metrics (e.g., DeepSeek-V3.1 accuracy 77.09% vs. 59.68%). Even in an online protocol where task i sees only tasks 1..i-1, the absence of a stated task ordering, combined with possible near-duplicate queries across splits, confounds comparison to static baselines. The paper must document the memory construction/evaluation split and report cold-start or disjoint-memory runs to rule out memorization.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"GeoForge is a training-free, self-evolving agent framework for Earth-observation (EO) tasks. It maintains three nonparametric memory structures—a Workflow Graph Memory, an Action-Level Experience Bank, and an Adapted Skill SOP—that are retrieved and injected as planning priors at inference time. After each episode, a safety-gated distillation step (Eqs. 16–18) converts grounded trajectories into updates to these memories without modifying backbone LLM weights. The paper reports experiments on Earth-Bench, ThinkGeo, and GeoPlan-Bench across several LLM backbones, claiming consistent improvements in task accuracy and tool-use trajectory quality, with ablations and a case study illustrating reduced redundant tool calls.","tokens_in":12634,"tokens_out":3566,"duration_ms":35292,"significance":"If the reported results hold, GeoForge is a meaningful contribution to Earth-observation agents and to the broader line of nonparametric, self-evolving agents. The framework is clearly specified: the three-level memory decomposition, the retrieval equations, and the safety gate are explicit, and the ablation study isolates each component's contribution. The training-free design and the emphasis on trajectory quality rather than only final accuracy are strengths. The major limitation is that the central empirical claim depends on a memory/evaluation split that is never documented; without this, the reported gains could be inflated by leakage of test-task information into the retrieved priors. The lack of variance estimates and the absence of code/data also weaken the current evidence.","major_comments":[{"comment":"The paper never states whether the trajectories used to populate the Workflow Graph Memory, Experience Bank, and SOP are disjoint from the tasks on which accuracy is reported. Eq. 18 updates memory from completed trajectories, and if those trajectories come from the same benchmark queries later scored, the retrieved priors can contain the exact tool sequences, parameter hints, and final answers, making the improvements in Tables 1–4 expected by construction. The authors must specify the memory construction/evaluation split, including whether memory is cold-started, pre-populated from training tasks, or grown online with an explicit task ordering, and they should report cold-start or disjoint-memory runs to rule out memorization.","section":"Experimental Setup — Evaluation Protocols"},{"comment":"All accuracy and trajectory metrics are reported as point estimates without error bars, confidence intervals, or significance tests. The claim that GeoForge 'consistently improves' task accuracy and trajectory quality across diverse backbones is therefore not statistically supported, especially for large gains such as DeepSeek-V3.1 accuracy rising from 59.68% to 77.09% and ThinkGeo answer accuracy rising from 9.78% to 60.98%. Please report multiple runs with variance and, where appropriate, significance tests, or explicitly state if these are single-run API evaluations.","section":"Tables 1–4 (Overall Performance)"},{"comment":"The central consistency claim is contradicted by the GPT-4o row: GeoForge's Tool-In-Order is 37.31, well below Earth-Agent's 53.20 and GeoEvolver's 48.93, even though the text states that trajectory-level metrics improve on most backbones. The abstract's 'consistently improves' phrasing is too strong given this counterexample. Please qualify the claim to state which metrics and backbones improve, and analyze why Tool-In-Order degrades for GPT-4o.","section":"Table 1, GPT-4o row"},{"comment":"Eqs. 3 and 7 introduce several free parameters (α, β, λC, λL, λE, L0, LS, Lmax, Rmax) and the sensitivity analysis varies only Top-k and Min-Retrieve-Score. The chosen defaults for all other parameters are not given, so it is unclear how sensitive the reported results are to these choices. Please provide the full hyperparameter settings and, if feasible, sensitivity analyses for the retrieval weights and thresholds.","section":"Sensitivity Analysis (Figure 5) and Implementation Details"}],"minor_comments":[{"comment":"The sentence 'In this paper, We presented GeoForge' has a capitalization typo; it should read 'we presented.'","section":"Conclusion"},{"comment":"The vertical axis is labeled only 'Score,' which is ambiguous; it should be labeled as Accuracy, Tool-Any-Order, Tool-In-Order, or Tool-Exact-Match, or the legend should be clarified.","section":"Figure 5"},{"comment":"MCP is used without defining the acronym on first use; please spell out 'Model Context Protocol.'","section":"Implementation Details"},{"comment":"Table 1 attributes OpenEarth-Agent to Dai et al. (2026), but the reference list associates OpenEarthAgent with Shabbir et al. (2026); please correct this citation.","section":"References"},{"comment":"The text repeatedly refers to 'the Appendix' for the full algorithm and additional metrics, but no appendix is included in the arXiv v1 file; please include it or state where it can be found.","section":"Appendix note"}],"recommendation":"major_revision","confidential_remarks":"The paper's central idea is promising, but the missing memory/evaluation split is the kind of issue that can be fixed with additional experiments or, if the split was already disjoint, by clearly documenting it. I would also ask the editor to verify the completeness of the appendix and the correctness of the citation for OpenEarthAgent. The absence of code or data is not, by itself, a reason for rejection, but it makes independent verification harder for a paper whose contribution is primarily empirical."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nGeoForge is a well-structured system paper with a real architectural contribution: a training-free, non-parametric memory layer for Earth observation agents, combining workflow graphs, action-level experiences, and an adapted skill SOP. The decomposition is sensible, the ablation study supports the contribution of each component, and the paper engages fairly with the existing self-evolving agent literature. If the reported gains hold up, this would be a meaningful advance for geospatial AI.\n\nBut there is a black box at the center of the empirical claims. The paper never states whether the trajectories used to build memory are disjoint from the evaluation tasks. The self-evolution loop (Eq. 18) updates memory from completed trajectories, and the evaluation protocol simply says they follow official benchmarks. If memory is warmed up on the same queries being scored, the retrieved priors can contain the exact tool sequences and answers, inflating both accuracy and trajectory metrics by construction. This is not a hypothetical concern; it is the load-bearing assumption behind every comparison table. The authors must document the split and report cold-start or disjoint-memory runs.\n\nThere is also an overclaim in the abstract: \"consistently improves both task accuracy and tool-use trajectory quality across diverse LLM backbones.\" Table 1 shows GPT-4o with lower accuracy than GeoEvolver (57.98 vs 65.59) and much lower Tool-In-Order (37.31 vs 48.93). \"Consistently\" is simply false. The lack of error bars and no code/data release are additional issues, and the appendix with the full algorithm is promised but not present in the preprint.\n\nThe math is heuristic, not deep, but this is a systems paper; that is fine. The citation pattern is solid, and the authors are not overselling novelty beyond the \"consistent\" slip.\n\nBottom line: the architecture deserves referee time. If the authors can show a clean evolution/evaluation split, this could be a solid paper. Without it, the empirical section is uninterpretable. I would send it to review but make the protocol question the first priority. I would not cite the numbers yet, but I would bring it to the reading group to discuss the evaluation pitfall.","headline":"GeoForge has a genuinely useful memory architecture, but the unstated evolution/evaluation split and the 'consistent improvement' overclaim make the current empirical claims untrustworthy.","tokens_in":13166,"tokens_out":3114,"would_cite":false,"duration_ms":26920,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GeoForge's three external memories improve Earth-observation agent accuracy and tool-use quality without retraining the backbone LLM.","keywords":["Earth observation agents","self-evolving agents","non-parametric memory","tool-use planning","workflow graphs","remote sensing","retrieval-augmented planning","LLM agents"],"falsifier":"Build GeoForge with memory populated only from a training split of a benchmark, run it on a held-out split, and report accuracy. If accuracy drops to the no-memory baseline level, the self-evolution claim is falsified; if it stays high, the claim survives. A cheaper check is to inspect the retrieved memory items for any that mention the test question IDs or their answers.","tokens_in":12131,"feed_emoji":"🛰️","tokens_out":5895,"duration_ms":51069,"temperature":0.7,"pith_summary":"The paper claims that an Earth-observation agent can keep improving at building scientifically valid tool workflows without any retraining of its underlying language model, by storing what past successful and failed trajectories teach in three complementary external memories. It argues that global operation order belongs in a workflow graph, local corrections belong in action-level experiences, and procedural and data constraints belong in an adapted skill standard operating procedure, and that retrieving these as a task-conditioned prior prevents the broken tool calls and tangled trajectories common in current EO agents. If this is right, then the bottleneck in reliable EO analysis is not the model's weights but how completed executions are distilled and reused, and agents can get substantially better at tasks like wildfire hotspot detection and spectral analysis simply by accumulating structured memory. The reported numbers support the claim: on one benchmark, accuracy rises from roughly 60 percent to 77 percent on a strong backbone, with the biggest error reductions in tool planning.","feed_headline":"External memory lifts Earth-observation agents to 77% accuracy","feed_subtitle":"Three self-evolving stores—workflow graphs, action lessons, skill SOPs—guide tool use with no model retraining.","key_machinery":"The carrying object is the non-parametric execution state: an external memory of three complementary stores—Workflow Graph Memory (a directed graph of tool sequences and reliability statistics), Action-Level Experiences (condition–action corrections), and an Adapted Skill SOP (procedural and data constraints)—that together are retrieved as a task-conditioned prior to guide a ReAct-style tool loop, with a safety-gated distillation step deciding what can be written back into memory. The graph supplies order, the SOP supplies procedure, and experiences supply local fixes; the gate prevents low-quality or non-generalizable trajectories from polluting the state.","core_discovery":"GeoForge's central discovery is that a training-free self-evolution loop organized at three decision levels—global workflow order, local action corrections, and procedural skill—can transfer execution knowledge across Earth-observation tasks. A directed reliability-aware workflow graph stores order-preserving tool sequences with success/failure counts; an experience bank stores condition–action corrections; an Adapted Skill SOP stores task decompositions and data/parameter constraints. At inference, sensing-context filtering shrinks the tool space and a similarity-scored retrieval assembles a compact execution context, but current observations remain the authority for the final answer. After each task, a safety gate checks that the trajectory used tools, produced a complete answer, did not exceed limits, and contains no forbidden patterns, and only then distills at most three lessons, one revised SOP, and two workflow candidates back into memory. The paper claims this loop, not better weights, is what improves accuracy and trajectory quality across backbones.","pith_inferences":["The paper evaluates on benchmarks where memory is distilled from executed trajectories; if those trajectories come from the same query set being scored, part of the gain may be memorization. A direct test would build memory only from a disjoint split and compare, which the paper does not report.","The same three-level memory pattern may transfer to other tool-use domains whose workflows have strict ordering and data-compatibility constraints, such as clinical lab pipelines or scientific data processing, though the retrievers would need domain-aware lexical projections.","The error-shift result suggests the next bottleneck is execution-trace and parameter errors; a future version might add memory entries specifically for parameter grounding, which the current SOP covers only as constraints."],"forward_implications":["If a backbone LLM is reused unchanged, an EO agent's planning can still improve episode by episode, so accuracy gains do not require expensive fine-tuning or large-scale retraining.","The three-level split predicts that different failure modes are fixed by different stores: removing the graph hurts trajectory order, removing the SOP hurts accuracy most, and removing experiences hurts local corrections, so memory design can be targeted to the dominant error type.","Because retrieval keeps final answers grounded in current observations rather than memory, the approach can work in settings where data changes over time, such as real-time wildfire or flood monitoring.","Reducing tool-planning and reasoning errors across most backbones implies the method is largely complementary to base-model capability, so stronger backbones should compound its gains."],"supporting_citations":[{"why":"Supplies the GeoEvolver baseline and the official Earth-Bench evaluation protocol that all main comparisons follow.","marker":"Dai et al. 2026"},{"why":"Introduced the Earth-Bench benchmark and the Earth-Agent baseline; most head-to-head accuracy and trajectory numbers come from this dataset.","marker":"Feng et al. 2025"},{"why":"Provided the ThinkGeo benchmark used to show gains in instruction alignment, tool accuracy, and grounded answer accuracy.","marker":"Shabbir et al. 2025"},{"why":"Contributed the GeoPlan-Bench benchmark and hierarchical task abstraction against which workflow planning results are scored.","marker":"Li et al. 2025"},{"why":"The ReAct reasoning-acting loop that GeoForge wraps its memory retrieval and tool execution around.","marker":"Yao et al. 2022"},{"why":"GeoEvolve is the prior self-evolving geospatial system whose limitation, evolving algorithms rather than whole workflows, motivates the three-level memory design.","marker":"Luo et al. 2025"}],"fun_headline_variants":["Memory-guided Earth-observation agents evolve without retraining","Training-free agent boosts EO reasoning via self-evolving memory","GeoForge: self-evolving tool workflows for satellite queries","Three-tier memory lifts Earth-observation agent accuracy","No retraining: agent adapts to Earth-observation tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains assume that the memory is built from trajectories that are not part of the evaluation set; the paper never states this split, and if the memory was seeded with the same tasks it is scored on, retrieval would be expected to inflate accuracy.","fun_headline_variants_meta":{"raw":{"variants":["Memory-guided Earth-observation agents evolve without retraining","Training-free agent boosts EO reasoning via self-evolving memory","GeoForge: self-evolving tool workflows for satellite queries","Three-tier memory lifts Earth-observation agent accuracy","No retraining: agent adapts to Earth-observation tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000908,"raw_usage":{"total_tokens":3917,"prompt_tokens":973,"completion_tokens":2944,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":589,"completion_tokens_details":{"reasoning_tokens":2866}},"tokens_in":589,"tokens_out":2944,"duration_ms":20600,"temperature":1.0,"reasoning_tokens":2866,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:19:04.684454+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build GeoForge with memory populated only from a training split of a benchmark, run it on a held-out split, and report accuracy. If accuracy drops to the no-memory baseline level, the self-evolution claim is falsified; if it stays high, the claim survives. A cheaper check is to inspect the retrieved memory items for any that mention the test question IDs or their answers.","supporting_citations":[],"review_version":1}