REVIEW 4 major objections 5 minor 46 references
GeoForge: Non-Parametric Self-Evolving Agents for Earth-Observation Reasoning
T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read GeoForge's three external memories improve Earth-observation agent accuracy and tool-use quality without retraining the backbone LLM.
desk verdict GeoForge has a genuinely useful memory architecture, but the unstated evolution/evaluation split and the 'consistent improvement' overclaim make the current empirical claims untrustworthy. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying object is the non-parametric execution state: an external memory of three complementary stores—Workflow Graph Memory (a directed graph of tool sequences and reliability statistics), Action-Level Experiences (condition–action corrections), and an Adapted Skill SOP (procedural and data constraints)—that together are retrieved as a task-conditioned prior to guide a ReAct-style tool loop, with a safety-gated distillation step deciding what can be written back into memory. The graph supplies order, the SOP supplies procedure, and experiences supply local fixes; the gate prevents low-quality or non-generalizable trajectories from polluting the state.
What would settle it
Build GeoForge with memory populated only from a training split of a benchmark, run it on a held-out split, and report accuracy. If accuracy drops to the no-memory baseline level, the self-evolution claim is falsified; if it stays high, the claim survives. A cheaper check is to inspect the retrieved memory items for any that mention the test question IDs or their answers.
Extended reading notes
Core claim
GeoForge's central discovery is that a training-free self-evolution loop organized at three decision levels—global workflow order, local action corrections, and procedural skill—can transfer execution knowledge across Earth-observation tasks. A directed reliability-aware workflow graph stores order-preserving tool sequences with success/failure counts; an experience bank stores condition–action corrections; an Adapted Skill SOP stores task decompositions and data/parameter constraints. At inference, sensing-context filtering shrinks the tool space and a similarity-scored retrieval assembles a compact execution context, but current observations remain the authority for the final answer. After each task, a safety gate checks that the trajectory used tools, produced a complete answer, did not exceed limits, and contains no forbidden patterns, and only then distills at most three lessons, one revised SOP, and two workflow candidates back into memory. The paper claims this loop, not better weights, is what improves accuracy and trajectory quality across backbones.
Load-bearing premise
The reported gains assume that the memory is built from trajectories that are not part of the evaluation set; the paper never states this split, and if the memory was seeded with the same tasks it is scored on, retrieval would be expected to inflate accuracy.
Editorial extensions
If this is right
- If a backbone LLM is reused unchanged, an EO agent's planning can still improve episode by episode, so accuracy gains do not require expensive fine-tuning or large-scale retraining.
- The three-level split predicts that different failure modes are fixed by different stores: removing the graph hurts trajectory order, removing the SOP hurts accuracy most, and removing experiences hurts local corrections, so memory design can be targeted to the dominant error type.
- Because retrieval keeps final answers grounded in current observations rather than memory, the approach can work in settings where data changes over time, such as real-time wildfire or flood monitoring.
- Reducing tool-planning and reasoning errors across most backbones implies the method is largely complementary to base-model capability, so stronger backbones should compound its gains.
Reading between the lines
- The paper evaluates on benchmarks where memory is distilled from executed trajectories; if those trajectories come from the same query set being scored, part of the gain may be memorization. A direct test would build memory only from a disjoint split and compare, which the paper does not report.
- The same three-level memory pattern may transfer to other tool-use domains whose workflows have strict ordering and data-compatibility constraints, such as clinical lab pipelines or scientific data processing, though the retrievers would need domain-aware lexical projections.
- The error-shift result suggests the next bottleneck is execution-trace and parameter errors; a future version might add memory entries specifically for parameter grounding, which the current SOP covers only as constraints.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. GeoForge is a training-free, self-evolving agent framework for Earth-observation (EO) tasks. It maintains three nonparametric memory structures—a Workflow Graph Memory, an Action-Level Experience Bank, and an Adapted Skill SOP—that are retrieved and injected as planning priors at inference time. After each episode, a safety-gated distillation step (Eqs. 16–18) converts grounded trajectories into updates to these memories without modifying backbone LLM weights. The paper reports experiments on Earth-Bench, ThinkGeo, and GeoPlan-Bench across several LLM backbones, claiming consistent improvements in task accuracy and tool-use trajectory quality, with ablations and a case study illustrating reduced redundant tool calls.
Significance. If the reported results hold, GeoForge is a meaningful contribution to Earth-observation agents and to the broader line of nonparametric, self-evolving agents. The framework is clearly specified: the three-level memory decomposition, the retrieval equations, and the safety gate are explicit, and the ablation study isolates each component's contribution. The training-free design and the emphasis on trajectory quality rather than only final accuracy are strengths. The major limitation is that the central empirical claim depends on a memory/evaluation split that is never documented; without this, the reported gains could be inflated by leakage of test-task information into the retrieved priors. The lack of variance estimates and the absence of code/data also weaken the current evidence.
major comments (4)
- [Experimental Setup — Evaluation Protocols] The paper never states whether the trajectories used to populate the Workflow Graph Memory, Experience Bank, and SOP are disjoint from the tasks on which accuracy is reported. Eq. 18 updates memory from completed trajectories, and if those trajectories come from the same benchmark queries later scored, the retrieved priors can contain the exact tool sequences, parameter hints, and final answers, making the improvements in Tables 1–4 expected by construction. The authors must specify the memory construction/evaluation split, including whether memory is cold-started, pre-populated from training tasks, or grown online with an explicit task ordering, and they should report cold-start or disjoint-memory runs to rule out memorization.
- [Tables 1–4 (Overall Performance)] All accuracy and trajectory metrics are reported as point estimates without error bars, confidence intervals, or significance tests. The claim that GeoForge 'consistently improves' task accuracy and trajectory quality across diverse backbones is therefore not statistically supported, especially for large gains such as DeepSeek-V3.1 accuracy rising from 59.68% to 77.09% and ThinkGeo answer accuracy rising from 9.78% to 60.98%. Please report multiple runs with variance and, where appropriate, significance tests, or explicitly state if these are single-run API evaluations.
- [Table 1, GPT-4o row] The central consistency claim is contradicted by the GPT-4o row: GeoForge's Tool-In-Order is 37.31, well below Earth-Agent's 53.20 and GeoEvolver's 48.93, even though the text states that trajectory-level metrics improve on most backbones. The abstract's 'consistently improves' phrasing is too strong given this counterexample. Please qualify the claim to state which metrics and backbones improve, and analyze why Tool-In-Order degrades for GPT-4o.
- [Sensitivity Analysis (Figure 5) and Implementation Details] Eqs. 3 and 7 introduce several free parameters (α, β, λC, λL, λE, L0, LS, Lmax, Rmax) and the sensitivity analysis varies only Top-k and Min-Retrieve-Score. The chosen defaults for all other parameters are not given, so it is unclear how sensitive the reported results are to these choices. Please provide the full hyperparameter settings and, if feasible, sensitivity analyses for the retrieval weights and thresholds.
minor comments (5)
- [Conclusion] The sentence 'In this paper, We presented GeoForge' has a capitalization typo; it should read 'we presented.'
- [Figure 5] The vertical axis is labeled only 'Score,' which is ambiguous; it should be labeled as Accuracy, Tool-Any-Order, Tool-In-Order, or Tool-Exact-Match, or the legend should be clarified.
- [Implementation Details] MCP is used without defining the acronym on first use; please spell out 'Model Context Protocol.'
- [References] Table 1 attributes OpenEarth-Agent to Dai et al. (2026), but the reference list associates OpenEarthAgent with Shabbir et al. (2026); please correct this citation.
- [Appendix note] The text repeatedly refers to 'the Appendix' for the full algorithm and additional metrics, but no appendix is included in the arXiv v1 file; please include it or state where it can be found.
Circularity Check
No in-text circular reduction found; the unspecified memory/evaluation split is an evaluation-protocol ambiguity, not a demonstrated circular step.
full rationale
GeoForge is an empirical framework paper: its central claim is that retrieving from workflow-graph, action-experience, and SOP memories improves EO accuracy and trajectory quality. The memory update loop (Eqs. 16-18) is the method under test, not a hidden reuse of the evaluation target; the retrieval scorers (Eqs. 3-8) are transparent heuristics rather than fitted parameters later renamed as predictions. The only self-citations (Xiao et al. 2025; Wang et al. 2026b) are background context and carry no load-bearing weight. The paper does not state whether the trajectories used to update memory (Eq. 18) are disjoint from the queries scored in Tables 1-2; if they were not, the trajectory metrics could be inflated by retrieval of memorized tool sequences. But the paper nowhere asserts such warm-starting, and no equation equates the stored memory content with the scored answer. Under the requirement that circularity be exhibited as a specific reduction rather than inferred from a missing protocol statement, this ambiguity is a correctness/evaluation-rigor concern, not a demonstrated circularity.
Assumptions & free parameters
free parameters (10)
- alpha, beta (similarity weights) =
alpha + beta = 1, values not stated in main text
- lambda_C (task-type consistency weight) =
not stated in main text
- lambda_L (length penalty coefficient) =
not stated in main text
- L0 (workflow length threshold) =
not stated in main text
- lambda_E (experience task-consistency weight) =
not stated in main text
- TopK (retrieved experiences) =
3
- MinRetrieveScore (workflow retrieval threshold) =
0.6
- LS (SOP length bound) =
not stated in main text
- Lmax (tool-call limit) =
not stated in main text
- Rmax (repetition limit) =
not stated in main text
assumptions (3)
- ad hoc to paper Retrieval based on tool-sequence Jaccard, lexical overlap, and task-type consistency is a sufficient prior for workflow quality.
- domain assumption The LLM distiller D_theta and safety gate psi produce accurate, generalizable, non-harmful memory updates without human supervision.
- domain assumption Memory evolution episodes are disjoint from evaluation tasks.
invented entities (3)
-
Workflow Graph Memory
-
Action-Level Experience Bank
-
Adapted Skill SOP
Cite this review
Pith. "Pith review of GeoForge: Non-Parametric Self-Evolving Agents for Earth-Observation Reasoning." pith.science (2026). https://pith.science/paper/24DNUX3J
@misc{pith2026260810494,
author = {Pith},
title = {Pith review of: GeoForge: Non-Parametric Self-Evolving Agents for Earth-Observation Reasoning},
year = {2026},
howpublished = {\url{https://pith.science/paper/24DNUX3J}},
note = {Machine review of arXiv:2608.10494}
}
read the original abstract
Earth observation (EO) agents construct scientifically valid tool workflows and ground their conclusions in current geospatial evidence. This is challenging because EO workflows are constrained by sensing semantics, product dependencies, spatial and temporal compatibility, and parameter requirements. Existing agents often search a broad operation space for each query, while recent self-evolving systems do not fully organize heterogeneous EO trajectories into reusable knowledge across different decision levels. To solve this problem, we present GeoForge, a training-free, self-evolving framework that transforms completed trajectories into a structured nonparametric execution state. GeoForge constrains the operation space according to the sensing context, then retrieves a task-conditioned prior from three complementary memories. Workflow Graph Memory captures global operation order, Action-Level Experiences provide local corrections, and the Adapted Skill Standard Operating Procedure preserves procedural and data constraints. The retrieved prior guides tool execution, while current observations remain the basis of the final answer. After each task, a safety-gated distillation process converts grounded trajectories into reusable execution knowledge for future retrieval. This execution, distillation, and reuse loop improves planning without updating the backbone LLM. Experiments on multiple geospatial benchmarks demonstrate that GeoForge consistently improves both task accuracy and tool-use trajectory quality across diverse LLM backbones, while substantially reducing tool-planning and reasoning errors for most LLMs.
Figures
Reference graph
Works this paper leans on
-
[1]
Advances in neural information processing systems , volume=
Chain-of-thought prompting elicits reasoning in large language models , author=. Advances in neural information processing systems , volume=
-
[2]
Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers) , pages=
Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models , author=. Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers) , pages=
-
[3]
and Mordatch, Igor , title =
Du, Yilun and Li, Shuang and Torralba, Antonio and Tenenbaum, Joshua B. and Mordatch, Igor , title =. Proceedings of the 41st International Conference on Machine Learning , articleno =. 2024 , publisher =
2024
-
[4]
International Conference on Learning Representations , volume=
Aflow: Automating agentic workflow generation , author=. International Conference on Learning Representations , volume=
-
[5]
arXiv preprint arXiv:2602.02559 , year=
Experience-Driven Multi-Agent Systems Are Training-free Context-aware Earth Observers , author=. arXiv preprint arXiv:2602.02559 , year=
-
[6]
Proceedings of the AAAI Conference on Artificial Intelligence , volume=
Expel: Llm agents are experiential learners , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=
-
[7]
Proceedings of the ACM Web Conference 2026 , pages=
Deepagent: A general reasoning agent with scalable toolsets , author=. Proceedings of the ACM Web Conference 2026 , pages=
work page 2026
-
[8]
arXiv preprint arXiv:2510.08191 , year=
Training-free group relative policy optimization , author=. arXiv preprint arXiv:2510.08191 , year=
Show all 46 references
-
[9]
International Conference on Learning Representations , volume=
MetaGPT: Meta programming for a multi-agent collaborative framework , author=. International Conference on Learning Representations , volume=
-
[10]
arXiv preprint arXiv:2505.02024 , year=
From mind to machine: The rise of manus ai as a fully autonomous digital agent , author=. arXiv preprint arXiv:2505.02024 , year=
-
[11]
arXiv preprint arXiv:2303.08774 , year=
Gpt-4 technical report , author=. arXiv preprint arXiv:2303.08774 , year=
-
[12]
Exploring the power of Coze’s no-code platform , author=
-
[13]
IGARSS 2024-2024 IEEE International Geoscience and Remote Sensing Symposium , pages=
Remote sensing chatgpt: Solving remote sensing tasks with chatgpt and visual models , author=. IGARSS 2024-2024 IEEE International Geoscience and Remote Sensing Symposium , pages=. 2024 , organization=
2024
-
[14]
arXiv preprint arXiv:2406.07089 , year=
RS-Agent: automating remote sensing tasks through intelligent agent , author=. arXiv preprint arXiv:2406.07089 , year=
-
[15]
arXiv preprint arXiv:2505.23752 , year=
Thinkgeo: Evaluating tool-augmented agents for remote sensing tasks , author=. arXiv preprint arXiv:2505.23752 , year=
-
[16]
arXiv preprint arXiv:2511.17198 , year=
Designing domain-specific agents via hierarchical task abstraction mechanism , author=. arXiv preprint arXiv:2511.17198 , year=
-
[17]
arXiv preprint arXiv:2602.17665 , year=
Openearthagent: A unified framework for tool-augmented geospatial agents , author=. arXiv preprint arXiv:2602.17665 , year=
-
[18]
arXiv preprint arXiv:2606.13148 , year=
TerraBench: Can Agents Reason Over Heterogeneous Earth-System Data? , author=. arXiv preprint arXiv:2606.13148 , year=
-
[19]
Proceedings of the ACM SIGCAS/SIGCHI Conference on Computing and Sustainable Societies , pages=
Geo-olm: Enabling sustainable earth observation studies with cost-efficient open language models & state-driven workflows , author=. Proceedings of the ACM SIGCAS/SIGCHI Conference on Computing and Sustainable Societies , pages=
-
[20]
arXiv preprint arXiv:2509.23141 , year=
Earth-agent: Unlocking the full landscape of earth observation with agents , author=. arXiv preprint arXiv:2509.23141 , year=
-
[21]
arXiv preprint arXiv:2607.12497 , year=
TerraLogic: A Benchmark for Hierarchical Geospatial Reasoning in Earth Observation , author=. arXiv preprint arXiv:2607.12497 , year=
-
[22]
arXiv preprint arXiv:2210.03629 , year=
React: Synergizing reasoning and acting in language models , author=. arXiv preprint arXiv:2210.03629 , year=
-
[23]
arXiv preprint arXiv:2404.14387 , year=
A survey on self-evolution of large language models , author=. arXiv preprint arXiv:2404.14387 , year=
-
[24]
arXiv preprint arXiv:2508.16153 , year=
Memento: Fine-tuning llm agents without fine-tuning llms , author=. arXiv preprint arXiv:2508.16153 , year=
-
[25]
arXiv preprint arXiv:2603.18743 , year=
Memento-skills: Let agents design agents , author=. arXiv preprint arXiv:2603.18743 , year=
-
[26]
arXiv preprint arXiv:2602.02474 , year=
MemSkill: Learning and Evolving Memory Skills for Self-Evolving Agents , author=. arXiv preprint arXiv:2602.02474 , year=
-
[27]
arXiv preprint arXiv:2603.10600 , year=
Trajectory-informed memory generation for self-improving agent systems , author=. arXiv preprint arXiv:2603.10600 , year=
-
[28]
arXiv preprint arXiv:2605.12061 , year=
SAGE: A Self-Evolving Agentic Graph-Memory Engine for Structure-Aware Associative Memory , author=. arXiv preprint arXiv:2605.12061 , year=
-
[29]
arXiv preprint arXiv:2603.12056 , year=
Xskill: Continual learning from experience and skills in multimodal agents , author=. arXiv preprint arXiv:2603.12056 , year=
-
[30]
IEEE Geoscience and Remote Sensing Magazine , volume=
Vision foundation models in remote sensing: A survey , author=. IEEE Geoscience and Remote Sensing Magazine , volume=. 2025 , publisher=
2025
-
[31]
Advances in Neural Information Processing Systems , volume=
Reflexion: Language Agents with Verbal Reinforcement Learning , author=. Advances in Neural Information Processing Systems , volume=
-
[32]
arXiv preprint arXiv:2308.08155 , year=
Autogen: Enabling next-gen llm applications via multi-agent conversation , author=. arXiv preprint arXiv:2308.08155 , year=
-
[33]
arXiv e-prints , pages=
A survey of self-evolving agents: On path to artificial super intelligence , author=. arXiv e-prints , pages=
-
[34]
2026 , publisher=
A systematic survey of self-evolving agents: From model-centric to environment-driven co-evolution , author=. 2026 , publisher=
2026
-
[35]
IEEE Geoscience and Remote Sensing Magazine , year=
Foundation models for remote sensing and earth observation: A survey , author=. IEEE Geoscience and Remote Sensing Magazine , year=
-
[36]
International Journal of Digital Earth , volume=
GPT, large language models (LLMs) and generative artificial intelligence (GAI) models in geospatial science: a systematic review , author=. International Journal of Digital Earth , volume=. 2024 , publisher=
2024
-
[37]
Journal of Electronics & Information Technology , volume=
A Large-Scale Multimodal Instruction Dataset for Remote Sensing Agents , author=. Journal of Electronics & Information Technology , volume=. 2026 , publisher=
2026
-
[38]
2025 , publisher=
Opportunities and challenges of integrating geographic information science and large language models , author=. 2025 , publisher=
2025
-
[39]
arXiv preprint arXiv:2501.09136 , year=
Agentic retrieval-augmented generation: A survey on agentic rag , author=. arXiv preprint arXiv:2501.09136 , year=
-
[40]
arXiv preprint arXiv:2512.13564 , year=
Memory in the age of ai agents , author=. arXiv preprint arXiv:2512.13564 , year=
-
[41]
International Journal of Geographical Information Science , pages=
GeoAgent: a hierarchical LLM-based multi-agent architecture for autonomous spatial analysis , author=. International Journal of Geographical Information Science , pages=. 2026 , publisher=
2026
-
[42]
Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
Geochat: Grounded large vision-language model for remote sensing , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
-
[43]
Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pages=
Automating Geospatial Vision Tasks with a Large Language Model Agent , author=. Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pages=. 2025 , organization=
2025
-
[44]
Advances in neural information processing systems , volume=
Mpnet: Masked and permuted pre-training for language understanding , author=. Advances in neural information processing systems , volume=
-
[45]
2025 , eprint=
GeoEvolve: Automating Geospatial Model Discovery via Multi-Agent Large Language Models , author=. 2025 , eprint=
2025
-
[46]
IEEE Geoscience and Remote Sensing Magazine , volume=
Deep learning in remote sensing: A comprehensive review and list of resources , author=. IEEE Geoscience and Remote Sensing Magazine , volume=. 2017 , publisher=
2017
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.