Pith. sign in

REVIEW 4 major objections 5 minor 61 references

AutoGEEval: A Multimodal and Automated Framework for Geospatial Code Generation on GEE with Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read AutoGEEval is an automated benchmark for measuring whether large language models can write Google Earth Engine code, tested on 1,325 API functions across 18 models.

desk verdict AutoGEEval ships a genuinely reusable execution-based benchmark and judge for GEE code generation, but the efficiency table has a copy-paste error and the Qwen-written oracle needs an audit before the rankings are trusted. read the letter →

arxiv 2505.12900 v1 pith:4UBUCTUF submitted 2025-05-19 cs.SE cs.AIcs.CGcs.CLcs.DB

Shuyang Hou , Zhangxiao Shen , Huayi Wu , Jianyuan Liang , Haoyue Jiao , Yaxian Qing , Xiaopu Zhang , Xu Li
show 3 more authors
Zhipeng Gui Xuefeng Guan Longgang Xiang
This is my paper · ORCID
classification cs.SEcs.AIcs.CGcs.CLcs.DB
keywords geospatialcodegenerationGoogleEarthEnginelargelanguagemodelsautomatedevaluationunittestbenchmarkLLMremotesensing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that geospatial code generation lacks a reproducible, automated measuring stick, and that AutoGEEval supplies one. It introduces a benchmark of 1,325 unit-level test cases, one per valid function in the Google Earth Engine client API documentation, spanning 26 output data types from images and geometries to lists and strings. The framework runs a model's generated function body with fixed parameters, executes it against the GEE Python API, and a Judge Program compares the saved output with a stored expected answer. Applied to 18 large language models, it reports pass@1 scores ranging from 31.40% to 71.55%, with parameter errors as the dominant failure mode. The paper concludes that current models' core weakness is domain-specific GEE knowledge, not general programming syntax.

What carries the argument

The load-bearing mechanism is the execution-and-compare loop. A test case stores the official documentation's function header plus a hidden reference implementation; the Submission Program prompts the target model to write only the function body, injects concrete parameters, runs the code on the GEE Python API, and writes the result to a designated path. The Judge Program then dispatches on the declared output type and compares the produced value to the expected answer using per-representation matching rules, with pass@n counting how often a correct output appears in n attempts. The framework also logs tokens, inference time, code lines, and runtime error categories, so the same loop produces cost and stability metrics rather than a single accuracy number.

What would settle it

Independently rewrite the expected answers for a random sample of AutoGEEval test cases from a different codebase or by GEE experts who have not seen the stored references, then rerun the 18 models through the Judge Program. If model pass rates shift materially, or if human experts disagree with the judge on a meaningful share of outputs, the framework's verdicts are not neutral.

Watch

Extended reading notes

Core claim

AutoGEEval is, the authors argue, the first multimodal unit-level automated evaluation framework for GEE code generation. Each test case is a six-tuple: function header, reference code, parameter list, output type, output path, and expected answer. During testing the reference code is hidden, parameters are injected, and the Judge Program maps the 26 declared GEE types onto runtime value representations—arrays, dictionaries, GeoJSON, lists, floats, and strings—applying a matching strategy for each; large images are validated by center sampling with a tolerance of 0.001. On this benchmark, pass@1 ranges from 31.40% to 71.55% and pass@5 reaches as high as 77.28%, while multiple generation attempts show sharply diminishing returns after pass@3. The paper reads the error logs as evidence that models have mature syntax skills but systematic gaps in GEE-specific parameter knowledge such as dataset identifiers, band names, and coordinate formats.

Load-bearing premise

The load-bearing premise is that the expected answers against which every model is judged are correct and unbiased; they were written by one large language model and then reviewed by human experts, so any error or stylistic convention baked into those reference answers would tilt the benchmark and the rankings.

Editorial extensions

If this is right

  • If AutoGEEval is right, future geospatial model evaluation can shift from expert manual scoring to an automated loop, making results reproducible across labs and models.
  • The error-type logs imply that targeted injection of GEE domain knowledge, such as dataset names, band names, and argument formats, should yield larger accuracy gains than further general code training, because parameter errors dominate failures.
  • The cost metrics imply that reporting accuracy alone is misleading: reasoning-augmented models can score competitively on pass@n while consuming several times more tokens and latency, and multi-sample gains diminish after the third attempt.
  • The 18-model comparison implies that model rank is not determined by category or size: a large general model leads, a code-specialized model from the same family is last, and the fine-tuned geospatial model improves over its base model by a modest margin.
  • The one-function-per-case scope means the measured capability is API-call correctness, not multi-step analytical workflow construction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: because the stored reference answers were drafted by one large language model and only human-reviewed, the benchmark may systematically favor outputs stylistically similar to that model's conventions, such as argument ordering, default values, or string formatting; an independent set of expert-written references could reorder the rankings.
  • My inference: the judge's center-sampling rule for large images means a model that computes the correct geospatial result but differs slightly at the sampled center could be marked wrong; a human re-scoring study of image-type outputs would quantify that risk.
  • My inference: the same framework could be extended to multi-function composition and cross-API tasks, and to retrieval-augmented variants, providing a controlled test of whether retrieval closes the parameter-error gap; this is a natural next experiment the paper does not run.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes AutoGEEval, an automated, unit-level evaluation framework for code generation on the Google Earth Engine (GEE) Python API. It constructs AutoGEEval-Bench, a set of 1,325 test cases covering 26 GEE output types, where each test case contains a function header, a reference code snippet, parameters, output type, output path, and an expected answer. A submission program prompts target LLMs to generate function bodies, executes them, and saves outputs; a judge program compares outputs with expected answers using type-specific matching strategies. The authors evaluate 18 LLMs (general, reasoning, code-specific, and geospatial) and report pass@1/pass@3/pass@5, coefficient of variation, stability-adjusted accuracy, token consumption, inference time, code lines, efficiency ratios, and error type distributions. The main claims are that AutoGEEval is the first multimodal unit-level automated evaluation framework for GEE code generation and that the results characterize model capabilities, with parameter errors being the dominant failure mode.

Significance. If the benchmark and judge are valid, AutoGEEval-Bench and the open-source framework constitute a valuable community resource for reproducible evaluation of geospatial code generation. The broad model comparison (18 models), the fine-grained error type analysis, and the inclusion of resource and efficiency metrics are useful contributions. The paper also ships the benchmark and code publicly (GitHub link in the data availability statement). However, the validity of the ground truth and the correctness of the efficiency tables are load-bearing for these contributions.

major comments (4)
  1. [Table 9 / Eqs. (10) and (12)] The In.T-E and Co.L-E columns in Table 9 are not computed according to Eqs. (10) and (12). For example, GPT-4o has Co.L-E = 7.77, which equals its Co.L value in Table 8 rather than pass@5/Co.L = 65.36/7.77 (approx. 8.41); DeepSeek-Coder-V2 has In.T-E = 8.39, which equals its In.T value rather than 40.75/8.39 (approx. 4.86). The error affects all locally deployed models for In.T-E and all models for Co.L-E, and consequently invalidates Co_Rank, E_Rank, and Total_Rank in Table 10. The authors must recompute these columns and the resulting rankings.
  2. [Section 3.3 / Table 5 / Eq. (6)] The ground truth (Expected_answer) is produced by executing reference code generated by Qwen2.5-Max, with expert revision reported only as 50 person-hours for 1,325 cases (about 2.3 minutes per case). The paper provides no per-case revision log, no inter-annotator agreement, and no audit of residual reference errors. Because six of the 18 evaluated models (Qwen-2.5-3B/7B/32B and Qwen2.5-Coder-3B/7B/32B) belong to the same model family as the oracle generator, any Qwen-specific conventions in the references (e.g., default arguments, band ordering, string formatting) could systematically disadvantage other models and bias the rankings. Please provide evidence that expert revision removed such conventions, e.g., a sample audit, a list of revised cases, or an independent re-verification subset.
  3. [Section 4.2 / Table 3] For ee.Image and ee.ImageCollection, the judge uses center sampling with a tolerance of 0.001 for large images. This means that correctness of the entire image is gated by a small central region. The paper does not report how many of the 224 ee.Image test cases fall into the 'large image' category, nor does it justify that center sampling is sufficient. If a functionally correct implementation differs from the reference only outside the sampled center (or within tolerance but in a spatially varying way), it will be marked wrong. Please quantify the impact and provide evidence that this approximation does not change the reported rankings.
  4. [Section 5.3.1 / Eq. (7)] The definition of pass@n is ambiguous and internally inconsistent. The text says N is the total number of generated samples and C_n is the number of incorrect samples, which would make pass@n independent of n; however, the reported pass@3 and pass@5 values are larger than pass@1, implying that C_n actually counts test cases for which all n attempts fail. Please define C_n and N precisely, state how the n samples per test case are aggregated, and explain whether this is the standard pass@k estimator or a different plug-in estimator.
minor comments (5)
  1. [Section 3.1 / Eq. (2)] The task definition states that code is generated 'within the Earth Engine JavaScript environment,' but the framework is implemented with the GEE Python API (Section 1). Please correct this inconsistency.
  2. [Table 3] The data type 'ee.Pixeltype' should be 'ee.PixelType' to match the GEE naming convention.
  3. [Table 10] QwQ-32B is assigned T_Rank 18 even though its Tok.-E is N/A. If a metric is unavailable, the ranking should exclude the model from that dimension rather than ranking it last, or the paper should justify the choice.
  4. [Section 6.1 / text near Figure 9] The reported average improvements (12.88% for pass@3, 3.81% for pass@5) and the ranges do not match Table 6. For example, the largest pass@3 improvement in Table 6 is 12.45 percentage points (DeepSeek-R1), not 21.37%, and the computed average of the 18 pass@3 improvements is approximately 6.6 percentage points. Please recompute these summary statistics.
  5. [Section 2.3] The related-work discussion acknowledges GeoSpatial-Code-LLMs but does not explicitly state how AutoGEEval's automation differs from it beyond sample size and data-type coverage; please make the comparison concrete.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark construction is an empirical validity concern, not a derivation that reduces to its own inputs.

full rationale

I examined the claimed derivation chain from AutoGEEval-Bench construction, through the Submission and Judge Programs, to the 18-model evaluation. The ground-truth answers are obtained by executing Reference_code generated by Qwen2.5-Max and then revised by five experts (Sections 3.2 and 3.3), and the Judge Program compares model outputs against those Expected_answer values. This is a benchmark-construction design with a plausible bias risk: correctness is defined relative to an LLM-written oracle, and some evaluated models belong to the same Qwen family. However, that is not circularity under the hard rules. There is no equation in which a predicted quantity is defined as the fitted input, no fitted parameter renamed as a prediction, no uniqueness theorem imported from the authors' prior work, and no ansatz smuggled in via self-citation. The related-work and GeoCode-GPT passages cite the authors' own earlier papers, but those citations are contextual and do not carry the load-bearing argument; the benchmark itself is built from official GEE documentation and executed code, and the model rankings are measured, not derived from the benchmark's assumptions. The absence of a per-case revision log and inter-annotator agreement is a real limitation and a threat to external validity, but it is a data-quality concern rather than a self-definitional or reductionist circular step. Therefore, under the specified rubric, the appropriate finding is no significant circularity, score 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the benchmark's validity, which depends on four classes of assumptions: the GEE documentation defines the tasks, the Qwen2.5-Max-written reference code defines the answers, the five-expert review caught reference errors at the reported effort, and GEE execution is deterministic. The model comparison additionally assumes API-hosted and local deployments are commensurable. No fitted physics-style parameters appear; the judge's image tolerance and center-sampling rule are hand-chosen thresholds that gate pass/fail, and temperature differs across model categories. The paper introduces no new physical or formal entities; SA and Total_Rank are composite indices, not postulated entities.

free parameters (3)
  • image comparison tolerance = 0.001
    Hand-chosen pixel-wise tolerance gates pass/fail for ee.Image and ee.ImageCollection outputs (Table 3); sampled pixels differing by more than 0.001 are marked wrong.
  • center-sampling rule for large images = center sample when image is large
    Hand-chosen strategy in Table 3: for large images the judge samples the center with tolerance 0.001, so correctness of the whole image is decided by the sampled region.
  • generation temperature = 0.2 for non-reasoning models; native default for reasoning models
    Generation settings differ across model categories (Section 5.2), which confounds accuracy and stability comparisons between categories.
assumptions (5)
  • domain assumption The official GEE Client Libraries documentation specifies intended function behavior, and the 1,374 documented functions map one-to-one to valid unit tests.
    Section 3.3: the benchmark is built from the official documentation; the paper assumes the docs are authoritative and complete for defining unit tasks.
  • ad hoc to paper Reference code generated by Qwen2.5-Max and executed produces the correct expected answer after expert review.
    Section 3.3: Reference_code is generated by Qwen2.5-Max from a prompt, and Expected_answer is the executed output of that code. The ground truth is an LLM-written oracle, not an independently derived specification.
  • domain assumption Five experts verified all 1,325 test cases at the reported effort of 50 hours.
    Section 3.3 and Table 5: expert revision is reported as 50 hours; whether that depth suffices to catch subtle reference-code errors is unverified.
  • domain assumption GEE execution is deterministic and consistent between reference runs and model runs.
    Sections 4.2 and 5: outputs are compared directly; if GEE results vary (image processing nondeterminism, server-side state), the fixed expected answers would misclassify.
  • domain assumption API-hosted and locally deployed models are comparable within one benchmark run.
    Section 5.2: proprietary models run via API and open models locally via Ollama; Section 6.2 itself notes that network latency skews inference-time comparisons.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AutoGEEval: A Multimodal and Automated Framework for Geospatial Code Generation on GEE with Large Language Models." pith.science (2026). https://pith.science/paper/4UBUCTUF

@misc{pith2026250512900,
  author       = {Pith},
  title        = {Pith review of: AutoGEEval: A Multimodal and Automated Framework for Geospatial Code Generation on GEE with Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4UBUCTUF}},
  note         = {Machine review of arXiv:2505.12900}
}
read the original abstract

Geospatial code generation is emerging as a key direction in the integration of artificial intelligence and geoscientific analysis. However, there remains a lack of standardized tools for automatic evaluation in this domain. To address this gap, we propose AutoGEEval, the first multimodal, unit-level automated evaluation framework for geospatial code generation tasks on the Google Earth Engine (GEE) platform powered by large language models (LLMs). Built upon the GEE Python API, AutoGEEval establishes a benchmark suite (AutoGEEval-Bench) comprising 1325 test cases that span 26 GEE data types. The framework integrates both question generation and answer verification components to enable an end-to-end automated evaluation pipeline-from function invocation to execution validation. AutoGEEval supports multidimensional quantitative analysis of model outputs in terms of accuracy, resource consumption, execution efficiency, and error types. We evaluate 18 state-of-the-art LLMs-including general-purpose, reasoning-augmented, code-centric, and geoscience-specialized models-revealing their performance characteristics and potential optimization pathways in GEE code generation. This work provides a unified protocol and foundational resource for the development and assessment of geospatial code generation models, advancing the frontier of automated natural language to domain-specific code translation.

Figures

Figures reproduced from arXiv: 2505.12900 by the authors.

Figure 1
Figure 1. Common error types in geospatial code generation with LLMs. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. AutoGEEval framework structure. The diagram highlights AutoGEEval-Bench, Submission Program, and Judge Program. Blue represents documentation, orange denotes language models, green represents prompts, and [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Example page from GEE API reference. After initial generation, all test cases were manually verified by a panel of five experts with extensive experience in GEE usage and geospatial code development. The verification process ensured that each test task reflects a valid geospatial analysis need, has a clear and accurate problem definition, and is configured with appropriate test inputs [PITH_FULL_IMAGE:figures/full_… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Prompt for unit test construction. For test cases that execute successfully and produce the expected results, the output is stored at the specified ‘output_path’ and serves as the ground-truth answer for that item. During the testing phase, the Judge Program retrieves …
Figure 7
Figure 7. Figure 7: Prompt for submission program 4.2. Judge program The overall workflow of the Judge Program is illustrated in Figure 2c. Its primary function is to read the execution results from the specified ‘Output_path’, select the appropriate evaluation logic based on the declared…
Figure 8
Figure 8. Figure 8: Stacked bar chart of pass@n metrics. The blue represents the pass@1 value, the orange represents the improvement of pass@3 over pass@1, and the green represents the improvement of pass@5 over pass@3. The white text on the bars indicates the absolute scores for pass@1, …
Figure 9
Figure 9. Figure 9: Line chart of pass@3 and pass@5 improvement ratios. The bubble chart displaying the pass@n scores and relative rankings of all models is shown in [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: LLM pass@n ranking bubble chart. The x-axis represents the pass@1 scores, the y-axis represents the pass@3 scores, and the size of the bubbles corresponds to the pass@5 scores. Different colors represent different LLM types, as shown in the legend. The bold and underl…
Figure 11
Figure 11. Figure 11: Average token consumption across LLMs [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]
Figure 12
Figure 12. Figure 12: Average inference time comparison of LLMs [PITH_FULL_IMAGE:figures/full_fig_p023_12.png]
Figure 13
Figure 13. Figure 13: Average lines of generated GEE code per model [PITH_FULL_IMAGE:figures/full_fig_p023_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

61 extracted references · 43 canonical work pages

  1. [1]

    Competition-level code generation with alphacode

    Li, Y .; Choi, D.; Chung, J.; Kushman, N.; Schrittwieser, J.; Leblond, R.; Eccles, T.; Keeling, J.; Gimeno, F.; Dal Lago, A. Competition-level code generation with alphacode. Science 2022, 378, 1092-1097

    work page 2022

  2. [2]

    Learning to code or coding to learn? A systematic review

    Popat, S.; Starkey, L. Learning to code or coding to learn? A systematic review. Computers & Education 2019, 128, 365-376

    work page 2019

  3. [3]

    An overview of transaction logic

    Bonner, A.J.; Kifer, M. An overview of transaction logic. Theoretical Computer Science 1994, 133, 205-265

    work page 1994

  4. [4]

    A survey on large language models for code generation

    Jiang, J.; Wang, F.; Shen, J.; Kim, S.; Kim, S. A survey on large language models for code generation. arXiv preprint arXiv:2406.00515 2024

    arXiv 2024

  5. [5]

    A review on code generation with llms: Application and evaluation

    Wang, J.; Chen, Y . A review on code generation with llms: Application and evaluation. 2023; pp. 284-289

    work page 2023

  6. [6]

    DeepSeek-Coder: When the Large Language Model Meets Programming --The Rise of Code Intelligence

    Guo, D.; Zhu, Q.; Yang, D.; Xie, Z.; Dong, K.; Zhang, W.; Chen, G.; Bi, X.; Wu, Y .; Li, Y .K. DeepSeek-Coder: When the Large Language Model Meets Programming --The Rise of Code Intelligence. arXiv preprint arXiv:2401.14196 2024

    arXiv 2024

  7. [7]

    Hui, B.; Yang, J.; Cui, Z.; Yang, J.; Liu, D.; Zha ng, L.; Liu, T.; Zhang, J.; Yu, B.; Lu, K. Qwen2. 5 -coder technical report. arXiv preprint arXiv:2409.12186 2024

    arXiv 2024

  8. [8]

    Code llama: Open foundation models for code

    Roziere, B.; Gehring, J.; Gloeckle, F.; Sootla, S.; Gat, I.; Tan, X.E.; Adi, Y .; Liu, J.; Sauvestre, R.; Remez, T. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 2023

    arXiv 2023

Show all 61 references
  1. [9]

    Code Hallucination

    Rahman, M.M.; Kundu, A. Code Hallucination. arXiv preprint arXiv:2407.04831 2024

    2024 arXiv

  2. [10]

    HumanEval on Latest GPT Models--2024

    Li, D.; Murr, L. HumanEval on Latest GPT Models--2024. arXiv preprint arXiv:2402.14852 2024

    2024 arXiv

  3. [11]

    HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation

    Yu, Z.; Zhao, Y .; Cohan, A.; Zhang, X.-P. HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation. arXiv preprint arXiv:2412.21199 2024

    2024 arXiv

  4. [12]

    -D.; Yan, F.; Zhang, T.; Wang, S.; Solar -Lezama, A.; Sen, K.; Stoica, I

    Jain, N.; Han, K.; Gu, A.; Li, W. -D.; Yan, F.; Zhang, T.; Wang, S.; Solar -Lezama, A.; Sen, K.; Stoica, I. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974 2024

    2024 arXiv

  5. [13]

    Bioconductor: open software development for computational biology and bioinformatics

    Gentleman, R.C.; Carey, V .J.; Bates, D.M.; Bolstad, B.; Dettling, M.; Dudoit, S.; Ellis, B.; Gautier, L.; Ge, Y .; Gentry, J. Bioconductor: open software development for computational biology and bioinformatics. Genome biology 2004, 5, 1-16

    2004

  6. [14]

    Computational finance using QuantLib-Python

    Varma, J.R.; Virmani, V . Computational finance using QuantLib-Python. Computing in Science & Engineering 2016, 18, 78-88

    2016

  7. [15]

    Orchestrating high-throughput genomic analysis with Bioconductor

    Huber, W.; Carey, V .J.; Gentleman, R.; Anders, S.; Carlson, M.; Carvalho, B.S.; Bravo, H.C.; Davis, S.; Gatto, L.; Girke, T. Orchestrating high-throughput genomic analysis with Bioconductor. Nature methods 2015, 12, 115-121

    2015

  8. [16]

    Orchestrating single -cell analysis with Biocon ductor

    Amezquita, R.A.; Lun, A.T.L.; Becht, E.; Carey, V .J.; Carpp, L.N.; Geistlinger, L.; Marini, F.; Rue-Albrecht, K.; Risso, D.; Soneson, C. Orchestrating single -cell analysis with Biocon ductor. Nature methods 2020, 17, 137-145

    2020

  9. [17]

    Domain -specific language techniques for visual computing: a comprehensive study

    Shen, L.; Chen, X.; Liu, R.; Wang, H.; Ji, G. Domain -specific language techniques for visual computing: a comprehensive study. Archives of Computational Methods in Engineering 2021, 28, 3113-3134

    2021

  10. [18]

    On the effectiveness of large language models in domain -specific code generation

    Gu, X.; Chen, M.; Lin, Y .; Hu, Y .; Zhang, H.; Wan, C.; Wei, Z.; Xu, Y .; Wang, J. On the effectiveness of large language models in domain -specific code generation. ACM Transactions on Software Engineering and Methodology 2025, 34, 1-22

    2025

  11. [19]

    Extracting land cover data using GEE: A review of the classification indices

    Capolupo, A.; Monterisi, C.; Caporusso, G.; Tarantino, E. Extracting land cover data using GEE: A review of the classification indices. 2020; pp. 782-796

    2020

  12. [20]

    Google Earth Engine for geo-big data ap plications: A meta-analysis and systematic review

    Tamiminia, H.; Salehi, B.; Mahdianpari, M.; Quackenbush, L.; Adeli, S.; Brisco, B. Google Earth Engine for geo-big data ap plications: A meta-analysis and systematic review. ISPRS journal of photogrammetry and remote sensing 2020, 164, 152-170

    2020

  13. [21]

    Tangible User Interfaces (TUIs): a novel paradigm for GIS

    Ratti, C.; Wang, Y .; Ishii, H.; Piper, B.; Frenchman, D. Tangible User Interfaces (TUIs): a novel paradigm for GIS. Transactions in GIS 2004, 8, 407-421

    2004

  14. [22]

    Progress and trends in the application of Google Earth and Google Earth Engine

    Zhao, Q.; Yu, L.; Li, X.; Peng, D.; Zhang, Y .; Gong, P . Progress and trends in the application of Google Earth and Google Earth Engine. Remote Sens. 2021, 13, 3778

    2021

  15. [23]

    Google earth engine applications

    Mutanga, O.; Kumar, L. Google earth engine applications. 2019, 11, 591

    2019

  16. [24]

    GeoCode -GPT: A large language model for geospatial code generation

    Hou, S.; Shen, Z.; Zhao, A.; Liang, J.; Gui, Z.; Guan, X.; Li, R.; Wu, H. GeoCode -GPT: A large language model for geospatial code generation. International Journal of Applied Earth Observation and Geoinformation 2025, 104456

    2025

  17. [25]

    GEE-OPs: An Operator Knowledge Base for Geospatial Code Generation on the Google Earth Engine Platform Powered by Large Language Models

    Hou, S.; Liang, J.; Zhao, A.; Wu, H. GEE-OPs: An Operator Knowledge Base for Geospatial Code Generation on the Google Earth Engine Platform Powered by Large Language Models. arXiv preprint arXiv:2412.05587 2024

    2024 arXiv

  18. [26]

    Google Earth Engine and artificial intelligence (AI): a comprehensive review

    Yang, L.; Driscol, J.; Sarigai, S.; Wu, Q.; Chen, H.; Lippitt, C.D. Google Earth Engine and artificial intelligence (AI): a comprehensive review. Remote Sens. 2022, 14, 3253

    2022

  19. [27]

    Can large language models generate geospatial code? arXiv preprint arXiv:2410.09738 2024

    Hou, S.; Shen, Z.; Liang, J.; Zhao, A.; Gui, Z.; Li, R.; Wu, H. Can large language models generate geospatial code? arXiv preprint arXiv:2410.09738 2024

    2024 arXiv

  20. [28]

    Evaluation of Code LLMs on Geospatial Code Generation

    Gramacki, P.; Martins, B.; Szymański, P . Evaluation of Code LLMs on Geospatial Code Generation. arXiv preprint arXiv:2410.04617 2024

    2024 arXiv

  21. [29]

    Chain -of-Programming (CoP): Empowering Large Language Models for Geospatial Code Generation

    Hou, S.; Jiao, H.; Shen, Z.; Liang, J.; Zhao, A.; Zhang, X.; Wang, J.; Wu, H. Chain -of-Programming (CoP): Empowering Large Language Models for Geospatial Code Generation. arXiv preprint arXiv:2411.10753 2024

    2024 arXiv

  22. [30]

    GeoCode -GPT: A Large Language Model for Geospatial Code Generation Tasks

    Hou, S.; Shen, Z.; Zhao, A.; Liang, J.; Gui, Z.; Guan, X.; Li, R.; Wu, H. GeoCode -GPT: A Large Language Model for Geospatial Code Generation Tasks. arXiv preprint arXiv:2410.17031 2024

    2024 arXiv

  23. [31]

    Geo -FuB: A Method for Constructing an Operator -Function Knowledge Base for Geospatial Code Generation Tasks Using Large Language Models

    Hou, S.; Zhao, A.; Liang, J.; Shen, Z.; Wu, H. Geo -FuB: A Method for Constructing an Operator -Function Knowledge Base for Geospatial Code Generation Tasks Using Large Language Models. arXiv preprint arXiv:2410.20975 2024

    2024 arXiv

  24. [32]

    Big Data and cloud computing: innovation opportunit ies and challenges

    Yang, C.; Huang, Q.; Li, Z.; Liu, K.; Hu, F. Big Data and cloud computing: innovation opportunit ies and challenges. International Journal of Digital Earth 2017, 10, 13-53

    2017

  25. [33]

    Evaluating large language models on geospatial tasks: a multiple geospatial task benchmarking study

    Xu, L.; Zhao, S.; Lin, Q.; Chen, L.; Luo, Q.; Wu, S.; Ye, X.; Feng, H.; Du, Z. Evaluating large language models on geospatial tasks: a multiple geospatial task benchmarking study. International Journal of Digital Earth 2025, 18, 2480268

    2025

  26. [34]

    GeoAI: spatially explicit artificial intelligence techniques for geographic knowledge discovery and beyond

    Janowicz, K.; Gao, S.; McKenzie, G.; Hu, Y .; Bhaduri, B. GeoAI: spatially explicit artificial intelligence techniques for geographic knowledge discovery and beyond. 2020, 34, 625-636

    2020

  27. [35]

    An overview of the Canada geographic information system (CGIS); Lands Directorate Environment Canada Ottawa, ON, Canada: 1980

    Fisher, T.; MacDonald, C. An overview of the Canada geographic information system (CGIS); Lands Directorate Environment Canada Ottawa, ON, Canada: 1980

    1980

  28. [36]

    GRASS GIS: A multi -purpose open source GIS

    Neteler, M.; Bowman, M.H.; Landa, M.; Metz, M. GRASS GIS: A multi -purpose open source GIS. Environmental Modelling & Software 2012, 31, 124-130

    2012

  29. [37]

    GISc ript: Towards an interoperable geospatial scripting language for GIS programming

    Zhang, M.; Yue, P.; Guo, X. GISc ript: Towards an interoperable geospatial scripting language for GIS programming. 2014; pp. 1-5

    2014

  30. [38]

    Open geospatial software and data: A review of the current state and a perspective into the future

    Coetzee, S.; Ivánová, I.; Mitasova, H.; Brovelli, M.A. Open geospatial software and data: A review of the current state and a perspective into the future. ISPRS Int. J. Geo-Inf. 2020, 9, 90

    2020

  31. [39]

    Code-literacy for GIS librarians: A discussion of languages, use cases, and competencies

    White, P.; Powell, S. Code-literacy for GIS librarians: A discussion of languages, use cases, and competencies. Journal of Map & Geography Libraries 2019, 15, 45-67

    2019

  32. [40]

    Show me the code: spatial analysis and open source

    Rey, S.J. Show me the code: spatial analysis and open source. Journal of Geographical Systems 2009, 11, 191- 207

    2009

  33. [41]

    Open geospatial analytics with PySAL

    Rey, S.J.; Anselin, L.; Li, X.; Pahle, R.; Laura, J.; Li, W.; Koschinsky, J. Open geospatial analytics with PySAL. ISPRS Int. J. Geo-Inf. 2015, 4, 815-836

    2015

  34. [42]

    Validation of GCC optimizers through trace generation

    Kanade, A.; Sanyal, A.; K hedker, U.P. Validation of GCC optimizers through trace generation. Software: Practice and Experience 2009, 39, 611-639

    2009

  35. [43]

    An empirical evaluation of Lex/Yacc and ANTLR parser generation tools

    Ortin, F.; Quiroga, J.; Rodriguez -Prieto, O.; Garcia, M. An empirical evaluation of Lex/Yacc and ANTLR parser generation tools. Plos one 2022, 17, e0264326

    2022

  36. [44]

    Template-based model generation

    He, X.; Zhang, T.; Pan, M.; Ma, Z.; Hu, C.-J. Template-based model generation. Software & Systems Modeling 2019, 18, 2051-2092

    2019

  37. [45]

    Systematic mapping study of template-based code generation

    Syriani, E.; Luhunu, L.; Sahraoui, H. Systematic mapping study of template-based code generation. Computer Languages, Systems & Structures 2018, 52, 43-62

    2018

  38. [46]

    Deepcoder: Learning to write programs

    Balog, M.; Gaunt, A.L.; Brockschmidt, M.; Nowozin, S.; Tarlow, D. Deepcoder: Learning to write programs. arXiv preprint arXiv:1611.01989 2016

    2016 arXiv

  39. [47]

    Seq2sql: G enerating structured queries from natural language using reinforcement learning

    Zhong, V .; Xiong, C.; Socher, R. Seq2sql: G enerating structured queries from natural language using reinforcement learning. arXiv preprint arXiv:1709.00103 2017

    2017 arXiv

  40. [48]

    code2seq: Generating sequences from structured representations of code

    Alon, U.; Brody, S.; Levy, O.; Yahav, E. code2seq: Generating sequences from structured representations of code. arXiv preprint arXiv:1808.01400 2018

    2018 arXiv

  41. [49]

    Codebert: A pre-trained model for programming and natural languages

    Feng, Z.; Guo, D.; Tang, D.; Duan, N.; Feng, X.; Gong, M.; Shou, L.; Qin, B.; Liu, T.; Jiang, D. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155 2020

    2002 arXiv

  42. [50]

    Github copilot ai pair programmer: Asset or liability? Journal of Systems and Software 2023, 203, 111734

    Dakhel, A.M.; Majdinasab, V .; Nikanjam, A.; Khomh, F.; Desmarais, M.C.; Jiang, Z.M.J. Github copilot ai pair programmer: Asset or liability? Journal of Systems and Software 2023, 203, 111734

    2023

  43. [51]

    A GPT-enhanced framework on knowledge extraction and reuse for geographic analysis models in Google Earth Engine

    Liang, J.; Zhao, A.; Hou, S.; Jin, F.; Wu, H. A GPT-enhanced framework on knowledge extraction and reuse for geographic analysis models in Google Earth Engine. International Journal of Digital Earth 2024, 17, 2398063

    2024

  44. [52]

    Large language models meet nl2code: A survey

    Zan, D.; Chen, B.; Zhang, F.; Lu, D.; Wu, B.; Guan, B.; Wang, Y .; Lou, J.-G. Large language models meet nl2code: A survey. arXiv preprint arXiv:2212.09420 2022

    2022 arXiv

  45. [53]

    MapGPT: an autonomous framework for mapping by integrating large language model and cartographic tools

    Zhang, Y .; He, Z.; Li, J.; Lin, J.; Guan, Q.; Yu, W. MapGPT: an autonomous framework for mapping by integrating large language model and cartographic tools. Cartography and Geographic Information Science 2024, 51, 717-743

    2024

  46. [54]

    ShapefileGPT: A Multi-Agent Large Language Model Framework for Automated Shapefile Processing

    Lin, Q.; Hu, R.; Li, H.; Wu, S.; Li, Y .; Fang, K.; Feng, H.; Du, Z.; Xu, L. ShapefileGPT: A Multi-Agent Large Language Model Framework for Automated Shapefile Processing. arXiv preprint arXiv:2410.12376 2024

    2024 arXiv

  47. [55]

    GIS Copilot: towards an autonomous GIS agent for spatial analysis

    Akinboyewa, T.; Li, Z.; Ning, H.; Lessani, M.N. GIS Copilot: towards an autonomous GIS agent for spatial analysis. International Journal of Digital Earth 2025, 18, 2497489

    2025

  48. [56]

    A survey on evaluating large language models in code generation tasks

    Chen, L.; Guo, Q.; Jia, H.; Zeng, Z.; Wang, X.; Xu, Y .; Wu, J.; Wang, Y .; Gao, Q.; Wang, J. A survey on evaluating large language models in code generation tasks. arXiv preprint arXiv:2408.16498 2024

    2024 arXiv

  49. [57]

    Are static analysis violations really fixed? a closer look at realistic usage of sonarqube

    Marcilio, D.; Bonifácio, R.; Monteiro, E.; Canedo, E.; Luz, W.; Pinto, G. Are static analysis violations really fixed? a closer look at realistic usage of sonarqube. 2019; pp. 209-219

    2019

  50. [58]

    Enhancing Code Readability through Automated Consistent Formatting

    Kanoutas, T.; Karanikiotis, T.; Symeonidis, A.L. Enhancing Code Readability through Automated Consistent Formatting. Electronics 2024, 13, 2073

    2024

  51. [59]

    -S.; Khan, F.S

    Aleissaee, A.A.; Kumar, A.; Anwer, R.M.; Khan, S.; Cholakkal, H.; Xia, G. -S.; Khan, F.S. Transformers in remote sensing: A survey. Remote Sens. 2023, 15, 1860

    2023

  52. [60]

    Remote sensing platforms and sensors: A survey

    Toth, C.; Jóźków, G. Remote sensing platforms and sensors: A survey. ISPRS Journal of Photogrammetry and Remote Sensing 2016, 115, 22-36

    2016

  53. [61]

    Automating Geospatial Analysis Workflows Using ChatGPT-4

    Zhang, Q.; Gao, S. Automating Geospatial Analysis Workflows Using ChatGPT-4. 2024; pp. 715-716

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.