Pith. sign in

REVIEW 5 major objections 5 minor 46 references

GeoAnalystBench: A GeoAI benchmark for assessing large language models for spatial analysis workflow and code generation

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read GeoAnalystBench, a new 50-task benchmark of Python geospatial problems, shows proprietary LLMs like ChatGPT-4o-mini far outperform open-source models such as DeepSeek-R1-7B in generating valid workflows and code, and finds that…

desk verdict A genuinely useful GeoAI benchmark, but the headline open-vs-closed model gap is partly a formatting artifact; it deserves peer review with fixes. read the letter →

arxiv 2509.05881 v1 pith:P4TW2QUQ submitted 2025-09-07 cs.SE cs.AI

Qianheng Zhang , Song Gao , Chen Wei , Yibo Zhao , Ying Nie , Ziru Chen , Shijie Chen , Yu Su
show 1 more author
Huan Sun
This is my paper · ORCID
classification cs.SEcs.AI
keywords GeoAIlargelanguagemodelsspatialanalysisworkflowcodegenerationGISautomationbenchmarkBLEUreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper presents GeoAnalystBench, a benchmark of 50 Python-based geospatial analysis tasks, each paired with a GIS-expert-designed reference workflow and reference Python code. It uses this benchmark to test whether large language models can replicate expert spatial-analysis logic when asked to generate both the workflow and the code. The central finding is a systematic gap: proprietary models (ChatGPT-4o-mini, Claude 3.5 Sonnet, Gemini 1.5 Flash) produce valid workflows in 93–96% of cases with CodeBLEU scores near 0.39, while smaller open-source models such as DeepSeek-R1-7B reach only 48.5% validity and CodeBLEU of 0.272. A mid-size open model, Llama 3.1 8B, approaches proprietary-level validity (95.3%) but still trails on code quality. The paper argues this framework offers a reproducible way to track how close LLMs are to automating GIS workflows, with tasks involving spatial relationships and optimal site selection remaining the most challenging.

What carries the argument

The object that carries the argument is the benchmark's reference standard and its composite scoring. Each of the 50 tasks has a human-designed workflow of 3–10 numbered geoprocessing steps, written by GIS experts from tutorials and research workflows, plus a reference Python implementation and structured metadata (domain knowledge and dataset description). Workflow quality is scored by validity (whether the response contains a parseable, coherent step sequence), mean absolute deviation of step count against the expert reference (MAD), and semantic similarity of the step text using MiniLM sentence embeddings. Code quality is scored by CodeBLEU, a weighted combination of n-gram match, weighted n-gram match, syntax AST match, and data-flow match. The interplay of these metrics is what lets the paper claim both 'LLMs can follow a workflow structure' and 'LLM-generated code is syntactically sound but only partially logically correct.'

What would settle it

Run the same 50 tasks with paraphrased instructions that remove surface similarity to the tutorials, and also have a second independent team of GIS experts write alternative reference workflows; if the proprietary-versus-open ranking or the category difficulty ordering changes substantially under either change, the reported gap is an artifact of prompt memorization or reference-choice arbitrariness rather than a stable measure of spatial reasoning.

Watch

Extended reading notes

Core claim

The paper's central claim is that GeoAnalystBench—built from 50 real-world geospatial tasks with expert workflows, code, domain knowledge, and dataset descriptions—provides a valid measure of an LLM's ability to plan and code spatial analysis, and that this measure reveals a reliable performance hierarchy. On that benchmark, ChatGPT-4o-mini, Claude 3.5 Sonnet, and Gemini 1.5 Flash each exceed 93% workflow validity and achieve CodeBLEU scores of 0.36–0.39, whereas DeepSeek-R1-7B and CodeLlama-7B produce valid workflows only 48.5% and 32.7% of the time respectively. The authors further claim that adding domain knowledge and dataset descriptions to the prompt consistently improves proprietary models, and that the hardest task categories across all models are determining how places are related and finding the best locations and paths, because these require multi-step spatial reasoning and optimization rather than pattern recognition.

Load-bearing premise

The expert-designed workflows are treated as the correct decomposition of each task, so if those references are tutorial-specific, arbitrary in granularity, or inconsistently judged, the benchmark score measures stylistic agreement with one expert's choices rather than genuine geospatial competence.

Editorial extensions

If this is right

  • Proprietary LLMs are already reliable enough to serve as assistants for routine multi-step geoprocessing, with validity above 93% and moderately aligned code.
  • Open-source models will need GIS-specific fine-tuning or larger scale to close the gap; Llama 3.1 8B's 95.3% validity shows the path is feasible.
  • Research and benchmark effort should concentrate on spatial relationship and location-optimization tasks, where even top models fail most often.
  • Prompting with domain knowledge and dataset descriptions is a low-cost lever that consistently improves proprietary model outputs, supporting retrieval-augmented generation for GIS.
  • Because CodeBLEU's syntax scores are high but data-flow scores are moderate (average 0.274), generated code needs execution-level validation before real deployment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The evaluation compares 7–8B open models with much larger proprietary models, so the measured 'open versus proprietary' gap partly reflects model size; a more controlled comparison with similarly sized open and closed models would isolate openness as a factor.
  • The linear workflow assumption (3–10 sequential steps) leaves out parallel subtasks and branching workflows, which the authors acknowledge; extending the benchmark to directed acyclic graphs would likely change both MAD and the difficulty ordering of categories.
  • If the underlying tutorials are heavily represented in LLM training data, the benchmark may overstate memorization-based performance; a paraphrased or novel task suite would test whether the measured capabilities generalize.
  • The manual validity checking means annotator judgment is load-bearing; replacing it with an end-to-end execution pass/fail on the generated code's outputs would produce a stricter, more objective capability signal.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. GeoAnalystBench is a new benchmark of 50 Python-based spatial-analysis tasks curated from ESRI tutorials, university labs, and academic sources, each annotated with an expert workflow (3-10 steps), reference code, domain-knowledge notes, and dataset descriptions. Six LLMs — three proprietary (GPT-4o-mini, Claude-3.5-Sonnet, Gemini-1.5-Flash) and three open-source (Llama-3.1-8B, CodeLlama-7B, DeepSeek-R1-7B) — are prompted under four knowledge conditions (no extra info, domain knowledge, dataset description, both) to generate workflow graphs and ArcPy/Python code. Workflow outputs are scored on validity (extractable workflow list), mean absolute deviation (MAD) of step count relative to the human reference, and cosine text similarity; code outputs are scored with CodeBLEU and its components. Results show GPT-4o-mini at 95.0% validity and 0.390 CodeBLEU versus DeepSeek-R1-7B at 48.5% and 0.272, and that tasks requiring spatial-relationship reasoning and optimal-location selection are hardest. The paper claims a clear proprietary/open-source capability gap, answers three research questions about the gap, the value of domain knowledge and dataset descriptions, and code-generation subskills, and offers two qualitative case studies comparing expert and LLM workflows.

Significance. The benchmark fills a genuine gap: it targets multi-step geoprocessing workflow construction rather than isolated GIS question-answering or single-tool code generation, and it pairs natural-language instructions with expert-validated reference workflows and code. The public release of the dataset and the reusable evaluation harness are real assets, and the two case studies provide useful qualitative evidence on parameter defaults and workflow granularity. The comparative claims are plausible in direction — the two best-performing models are proprietary and the two worst are open-source — but, as detailed below, the validity metric is confounded by a contradictory output-format instruction, the open-versus-closed framing is weakened by Llama-3.1-8B's near-parity on validity, and the code-quality claims rest on unexecuted code. These issues are fixable within a revision, and the benchmark resource itself is valuable regardless of the ranking outcome.

major comments (5)
  1. [4.1, Appendices A.1/A.2/B.1/B.2] The workflow and code prompt templates are internally contradictory: Key Note 1 instructs the model to "Use automatic reasoning and clearly explain each step" while Key Note 5 demands "Limit your output to code, no extra information" (the code prompt has the same structure; see Appendices A.1 and A.2). Because validity (Section 4.1) is defined as containing an "extractable workflow list" via an unspecified "combination of automated parsing and manual checking," a reasoning-tuned model that emits chain-of-thought before the workflow list — as DeepSeek-R1-7B is designed to do — will be scored invalid even if its workflow is correct. The reported 48.5% validity for DeepSeek-R1-7B and the Section 5.1 conclusion that distillation reduces "reasoning depth and generalization capacity" are therefore potentially artifacts of output-format noncompliance rather than evidence about spatial competence. I verified the contradiction in the supplied templates; the manuscript reports no parser specification, no reannotation of invalid outputs for embedded workflows, and no inter-annotator agreement, all of which are needed before the headline gap can be trusted.
  2. [5.1, Table 2, abstract] The abstract's claim of "a clear gap" between proprietary and open-source models is not supported by Table 2: the open-source Llama-3.1-8B achieves 95.3% validity against 95.0% for ChatGPT-4o-mini and 96.0% for Gemini-1.5-Flash, so the validity gap is driven by DeepSeek-R1-7B (48.5%) and CodeLlama-7B (32.7%), not by open-source status in general; text similarity and CodeBLEU also show Llama within a moderate range of the proprietary models (0.39 vs 0.54-0.56 TS; 0.340 vs 0.358-0.390 CodeBLEU). The comparison further confounds open-source status with model scale, since all open-source models here are 7-8B parameters while the proprietary models are far larger (undisclosed). A 70B-class open model should be added, or the conclusions in Section 5.4 and the abstract must be reframed as model-scale and format-compliance findings rather than an open-versus-closed gap.
  3. [4.2, Eq. (1), 5.2, Table 3] CodeBLEU is reported without executing the generated code, and the manuscript's own discussion (Section 5.2) notes that "many LLM generated workflows fail to run due to syntax or semantic errors" without reporting execution rates. The statement that "the relatively high syntax score implies that most outputs are structurally correct" misreads the CodeBLEU Syntax AST Match, which measures overlap with the reference AST, not syntactic validity of the generated code; similarly, the data-flow component (average 0.274) is not a direct test of functional correctness. To support the code-quality claims underlying RQ3, the authors should report an execution-based success metric (as in the cited ScienceAgentBench) or explicitly limit the claims to stylistic alignment.
  4. [3.1, 4.1] Validity is assessed by "a combination of automated parsing and manual checking" (Section 4.1), but the manuscript provides no protocol details: how many validators judged the responses, whether they are the same experts who designed the reference workflows (a potential circularity), how disagreements were resolved, and what the inter-annotator agreement was. Without this protocol, the validity column of Table 2 — the load-bearing input to the abstract's headline numbers — is not independently reproducible. The limitation sentence in Section 8 ("requires clearer definitions and high-quality annotations") confirms the reporting gap rather than repairing it.
  5. [Tables 2-3, 5.2, 5.3] All headline comparisons are point estimates without uncertainty quantification: with 50 tasks and three samples per condition, differences such as 95.0% versus 95.3% validity and CodeBLEU 0.390 versus 0.340 are within plausible sampling noise. The claim that "ChatGPT-4o-mini leads with the best average score" (Section 5.2) and the category-level differences in Section 5.3 need bootstrap confidence intervals or a paired statistical test across the 50 tasks before any ranking conclusion is drawn.
minor comments (5)
  1. [A.1] The "Expected Sample Output" in the elk home-range case study shows a graph titled "Workflow for Analyzing Urban Heat Using Kriging Interpolation," which appears copied from another task; the sample should match the case-study task.
  2. [4.1, 5.1] The manuscript uses inconsistent terminology for the same quantity: "Mean Absolute Deviation" in Section 4.1 but "Absolute Mean Difference" in the Section 5.1 table caption and text; one term should be used throughout.
  3. [A.2, B.2] The Key Note "Use latest open source Arcpy packages only" is internally contradictory because ArcPy is a proprietary ESRI package, not an open-source package; this may have further confused models when following the output-format restrictions.
  4. [Tables S-1 to S-3] The benchmark contains several near-duplicate tasks on the same underlying problem (mountain lion corridors: IDs 12, 21, 22, 32, 45; land subsidence: IDs 7 and 33), so the effective number of independent tasks is below 50; this should be acknowledged when reporting task-level statistics.
  5. [6.2.3, references] There is a typo in the Section 6.2.3 heading ("Result Comparision"), a missing citation placeholder "(?)" in the CodeLlama reference, and inconsistent capitalization in several reference titles; a final proofreading pass is needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: GeoAnalystBench compares LLM outputs to externally curated human expert workflows; metrics are not fitted parameters and no claim reduces by construction to its own inputs.

full rationale

GeoAnalystBench is an evaluation artifact rather than a derived prediction. The reference workflows, tasks, and code are curated by GIS experts from tutorials and publications (Section 3.1), and the evaluation metrics — valid rate, workflow-length MAD, text similarity, and CodeBLEU — are computed against those external human references or off-the-shelf tools, not fitted to the model outputs. The abstract's headline comparison (ChatGPT-4o-mini 95% validity / 0.39 CodeBLEU vs DeepSeek-R1-7B 48.5% / 0.272) is an empirical measurement, not a quantity defined by the benchmark's own equations. The paper's self-citation to ScienceAgentBench (Chen et al., 2024, Section 3.1) only describes the prompt/knowledge format and is not load-bearing for any result. The manuscript itself flags the risk that tasks adapted from public tutorials may appear in LLM training data (Section 8) and argues the performance spread shows tasks are not memorized; this is a data-contamination concern, not circularity. The strongest caveat is construct validity: Section 4.1 defines validity as 'contains an extractable workflow list' while the prompt templates (Section 4.1 and Appendices A.1/A.2) demand both reasoning text and 'code only,' which could differentially depress reasoning-heavy open models such as DeepSeek-R1-7B. That is a possible measurement confound about what the benchmark measures, but it is not a reduction of the paper's claim to its own inputs, so it does not raise the circularity score.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The benchmark itself is a dataset, not a derivation; no fitted constants are introduced. The main assumptions concern the validity of human workflows as ground truth and the meaningfulness of the chosen metrics. No new entities are postulated.

free parameters (3)
  • sampling temperature = 0.7
    Hand-chosen for all models to balance consistency and diversity (Section 5); the ranking could shift at other temperatures.
  • output samples per prompt = 3
    Three outputs per prompt were collected and aggregated; no full distribution is reported beyond MAD standard deviations.
  • CodeBLEU weights = 0.2, 0.2, 0.3, 0.3
    Weights from Ren et al. (2020) are used as-is and not validated for geospatial code; they affect the final CodeBLEU ranking.
assumptions (5)
  • domain assumption Human expert workflows are the ground truth for correct spatial analysis task decomposition.
    Used throughout evaluation (Sections 3.1, 4.1); if reference workflows are not canonical, alignment scores are not meaningful.
  • domain assumption CodeBLEU scores correlate with code quality for geospatial Python code.
    CodeBLEU is applied as the primary code metric (Section 4.2) without validation against execution success or expert code review.
  • ad hoc to paper The 50 tasks sampled from ESRI tutorials and academic sources are representative of real-world GIS work.
    Task selection is heuristic; the paper itself acknowledges possible overlap with LLM training data (Section 8).
  • standard math The all-MiniLM-L6-v2 sentence transformer provides semantically meaningful embeddings for workflow step lists.
    Used for text similarity (Section 4.1); this is an external model, standard practice, but introduces its own embedding biases.
  • domain assumption LLM outputs are comparable under temperature 0.7 and three samples per prompt.
    No variance analysis across different seeds or sample sizes is reported, so the stability of the ranking is unknown.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GeoAnalystBench: A GeoAI benchmark for assessing large language models for spatial analysis workflow and code generation." pith.science (2026). https://pith.science/paper/P4TW2QUQ

@misc{pith2026250905881,
  author       = {Pith},
  title        = {Pith review of: GeoAnalystBench: A GeoAI benchmark for assessing large language models for spatial analysis workflow and code generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P4TW2QUQ}},
  note         = {Machine review of arXiv:2509.05881}
}
read the original abstract

Recent advances in large language models (LLMs) have fueled growing interest in automating geospatial analysis and GIS workflows, yet their actual capabilities remain uncertain. In this work, we call for rigorous evaluation of LLMs on well-defined geoprocessing tasks before making claims about full GIS automation. To this end, we present GeoAnalystBench, a benchmark of 50 Python-based tasks derived from real-world geospatial problems and carefully validated by GIS experts. Each task is paired with a minimum deliverable product, and evaluation covers workflow validity, structural alignment, semantic similarity, and code quality (CodeBLEU). Using this benchmark, we assess both proprietary and open source models. Results reveal a clear gap: proprietary models such as ChatGPT-4o-mini achieve high validity 95% and stronger code alignment (CodeBLEU 0.39), while smaller open source models like DeepSeek-R1-7B often generate incomplete or inconsistent workflows (48.5% validity, 0.272 CodeBLEU). Tasks requiring deeper spatial reasoning, such as spatial relationship detection or optimal site selection, remain the most challenging across all models. These findings demonstrate both the promise and limitations of current LLMs in GIS automation and provide a reproducible framework to advance GeoAI research with human-in-the-loop support.

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 28 canonical work pages

  1. [1]

    and Lessani, M

    Akinboyewa, T., Li, Z., Ning, H. and Lessani, M. N. (2025) Gis copilot: Towards an autonomous gis agent for spatial analysis. International Journal of Digital Earth, 18, 2497489

    work page 2025

  2. [2]

    E., Wulder, M

    Andrew, M. E., Wulder, M. A., Nelson, T. A. and Coops, N. C. (2015) Spatial data, analysis approaches, and information needs for spatial ecosystem service assessments: a review. GIScience & Remote Sensing, 52, 344--373

    work page 2015

  3. [3]

    ://www.anthropic.com/news/claude-3-5-sonnet

    Anthropic (2024) Claude 3.5 sonnet. ://www.anthropic.com/news/claude-3-5-sonnet. Accessed: 2025-02-12

    work page 2024

  4. [4]

    and Waters, N

    Cervone, G., Sava, E., Huang, Q., Schnebele, E., Harrison, J. and Waters, N. (2015) Using Twitter for tasking remote-sensing data collection and damage assessment: 2013 Boulder flood case study . International Journal of Remote Sensing, 37, 100--124. ://doi.org/10.1080/01431161.2015.1117684

    arXiv 2015

  5. [5]

    Chen, J., Tang, J., Qin, J., Liang, X., Liu, L., Xing, E. P. and Lin, L. (2021) GeOQA: A geometric question answering benchmark towards multimodal numerical Reasoning . arXiv (Cornell University). ://arxiv.org/abs/2105.14517

    arXiv 2021

  6. [6]

    Chen, Z., Chen, S., Ning, Y., Zhang, Q., Wang, B., Yu, B., Li, Y., Liao, Z., Wei, C., Lu, Z. et al. (2024) Scienceagentbench: Toward rigorous assessment of language agents for data-driven scientific discovery. arXiv preprint arXiv:2410.05080

    arXiv 2024

  7. [7]

    S., Munir, M

    Danish, M. S., Munir, M. A., Shah, S. R. A., Kuckreja, K., Khan, F. S., Fraccaro, P., Lacoste, A. and Khan, S. (2024) GEOBench-VLM: Benchmarking Vision-Language Models for geospatial tasks . ://arxiv.org/abs/2411.19325

    arXiv 2024

  8. [8]

    DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., Zhang, X., Yu, X., Wu, Y., Wu, Z. F., Gou, Z., Shao, Z., Li, Z., Gao, Z., Liu, A., Xue, B., Wang, B., Wu, B., Feng, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., Dai, D., Chen, D., Ji, D., Li, E., Lin, F., Dai, F., Luo, F., Hao, G., Chen, G., ...

    arXiv 2025

Show all 46 references
  1. [9]

    ESRI (2013) The language of spatial analysis . ESRI. ://www.esri.com/content/dam/esrisites/sitecore-archive/Files/Pdfs/library/books/the-language-of-spatial-analysis.pdf

    2013

  2. [10]

    and Szyma \'n ski, P

    Gramacki, P., Martins, B. and Szyma \'n ski, P. (2024) Evaluation of code llms on geospatial code generation. In Proceedings of the 7th ACM SIGSPATIAL International Workshop on AI for Geographic Knowledge Discovery, 54--62

    2024

  3. [11]

    Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gre...

    2024 arXiv

  4. [12]

    and Wu, H

    Hou, S., Liang, J., Zhao, A. and Wu, H. (2024) Gee-ops: An operator knowledge base for geospatial code generation on the google earth engine platform powered by large language models. arXiv preprint arXiv:2412.05587

    2024 arXiv

  5. [13]

    and Pfister, T

    Hsieh, C.-Y., Li, C.-L., Yeh, C.-K., Nakhost, H., Fujii, Y., Ratner, A., Krishna, R., Lee, C.-Y. and Pfister, T. (2023) Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. ://arxiv.org/abs/2305.02301

    2023 arXiv

  6. [14]

    and Cong, G

    Huang, W., Wang, J. and Cong, G. (2024) Zero-shot urban function inference with street view images through prompting a pretrained vision-language model. International Journal of Geographical Information Science, 38, 1414--1442

    2024

  7. [15]

    Jakubik, J., Roy, S., Phillips, C., Fraccaro, P., Godwin, D., Zadrozny, B., Szwarcman, D., Gomes, C., Nyirjesy, G., Edwards, B. et al. (2023) Foundation models for generalist geospatial artificial intelligence. arXiv preprint arXiv:2310.18660

    2023 arXiv

  8. [16]

    and Bhaduri, B

    Janowicz, K., Gao, S., McKenzie, G., Hu, Y. and Bhaduri, B. (2020) GeoAI: spatially explicit artificial intelligence techniques for geographic knowledge discovery and beyond . International Journal of Geographical Information Science, 34, 625--636. ://doi.org/10.1080/13658816....

    2020

  9. [17]

    and Cai, L

    Janowicz, K., Mai, G., Huang, W., Zhu, R., Lao, N. and Cai, L. (2025) GeoFM: how will geo-foundation models reshape spatial data science and GeoAI? International Journal of Geographical Information Science, 39, 1849--1865

    2025

  10. [18]

    and Janowicz, K

    Ji, Y., Gao, S., Nie, Y., Maji \'c , I. and Janowicz, K. (2025) Foundation models for geospatial reasoning: assessing the capabilities of large language models in understanding geometries and topological spatial relations. International Journal of Geographical Information Scie...

    2025

  11. [19]

    and Yang, C

    Jiang, Y. and Yang, C. (2024) Is chatgpt a good geospatial data analyst? exploring the integration of natural language into structured query language within a spatial database. ISPRS International Journal of Geo-Information, 13, 26

    2024

  12. [20]

    and Kochedykov, D

    Krechetova, V. and Kochedykov, D. (2025) GeoBenchX: Benchmarking LLMs for Multistep geospatial Tasks . ://arxiv.org/abs/2503.18129

    2025

  13. [21]

    u ttler, H., Lewis, M., Yih, W.-t., Rockt \

    Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., K \"u ttler, H., Lewis, M., Yih, W.-t., Rockt \"a schel, T. et al. (2020) Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33, 9459--9474

    2020

  14. [22]

    and He, X

    Li, D., Ding, R., Zhang, Q., Li, Z., Chen, B., Xie, P., Xu, Y., Li, X., Guo, N., Huang, F. and He, X. (2023) GeoGLUE: a GeoGraphic Language Understanding Evaluation Benchmark . arXiv (Cornell University). ://arxiv.org/abs/2305.06545

    2023 arXiv

  15. [23]

    and Ning, H

    Li, Z. and Ning, H. (2023) Autonomous gis: the next-generation ai-powered gis. International Journal of Digital Earth, 16, 4668--4686

    2023

  16. [24]

    T., Yang, C., Bhaduri, B., Wang, S., Zhu, A

    Li, Z., Ning, H., Gao, S., Janowicz, K., Li, W., Arundel, S. T., Yang, C., Bhaduri, B., Wang, S., Zhu, A. et al. (2025) Giscience in the era of artificial intelligence: A research agenda towards autonomous gis. arXiv preprint arXiv:2503.23633

    2025 arXiv

  17. [25]

    Mai, G., Huang, W., Sun, J., Song, S., Mishra, D., Liu, N., Gao, S., Liu, T., Cong, G., Hu, Y. et al. (2024) On the opportunities and challenges of foundation models for geoai (vision paper). ACM Transactions on Spatial Algorithms and Systems, 10, 1--46

    2024

  18. [26]

    and Lessani, M

    Ning, H., Li, Z., Akinboyewa, T. and Lessani, M. N. (2025) An autonomous gis agent framework for geospatial data retrieval. International Journal of Digital Earth, 18, 2458688

    2025

  19. [27]

    M., Yamaguchi, Y

    Pham, H. M., Yamaguchi, Y. and Bui, T. Q. (2011) A case study on the relation between city planning and urban growth using remote sensing and spatial metrics. Landscape and Urban Planning, 100, 223--230

    2011

  20. [28]

    and Ma, S

    Ren, S., Guo, D., Lu, S., Zhou, L., Liu, S., Tang, D., Sundaresan, N., Zhou, M., Blanco, A. and Ma, S. (2020) Codebleu: a method for automatic evaluation of code synthesis. arXiv preprint arXiv:2009.10297

    2020 arXiv

  21. [29]

    P., Galley, M., Caruana, R

    Singh, C., Inala, J. P., Galley, M., Caruana, R. and Gao, J. (2024) Rethinking interpretability in the era of large language models . ://arxiv.org/abs/2402.01761

    2024 arXiv

  22. [30]

    (2015) ArcPy and ArcGIS--Geospatial Analysis with Python

    Toms, S. (2015) ArcPy and ArcGIS--Geospatial Analysis with Python. Packt Publishing Ltd

    2015

  23. [31]

    and Ye, X

    Wang, S., Hu, T., Xiao, H., Li, Y., Zhang, C., Ning, H., Zhu, R., Li, Z. and Ye, X. (2024) GPT, large language models (LLMs) and generative artificial intelligence (GAI) models in geospatial science: a systematic review . International Journal of Digital Earth, 17, 2353122

    2024

  24. [32]

    and Zhou, M

    Wang, W., Wei, F., Dong, L., Bao, H., Yang, N. and Zhou, M. (2020) Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. ://arxiv.org/abs/2002.10957

    2020 arXiv

  25. [33]

    and Claramunt, C

    Van de Weghe, N., De Sloover, L., Cohn, A., Huang, H., Scheider, S., Sieber, R., Timpf, S. and Claramunt, C. (2025) Opportunities and challenges of integrating geographic information science and large language models. Journal of Spatial Information Science, 93--116

    2025

  26. [34]

    and Yu, W

    Wei, C., Zhang, Y., Zhao, X., Zeng, Z., Wang, Z., Lin, J., Guan, Q. and Yu, W. (2024) Geotool-gpt: a trainable method for facilitating large language models to master gis tools. International Journal of Geographical Information Science, 1--25

    2024

  27. [35]

    V., Zhou, D

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D. et al. (2022) Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35, 24824--24837

    2022

  28. [36]

    and Zhang, Z

    Wu, M., Huang, Q., Gao, S. and Zhang, Z. (2023) Mixed land use measurement and mapping with street view images and spatial context-aware prompts via zero-shot multimodal learning. International Journal of Applied Earth Observation and Geoinformation, 125, 103591

    2023

  29. [37]

    and Du, Z

    Xu, L., Zhao, S., Lin, Q., Chen, L., Luo, Q., Wu, S., Ye, X., Feng, H. and Du, Z. (2025) Evaluating large language models on geospatial tasks: a multiple geospatial task benchmarking study. International Journal of Digital Earth, 18, 2480268

    2025

  30. [38]

    Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H. et al. (2024) Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

    2024 arXiv

  31. [39]

    Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y. et al. (2024) Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and ...

    2024

  32. [40]

    and Gao, S

    Zhang, Q. and Gao, S. (2024) Automating geospatial analysis workflows using chatgpt-4. In Proceedings of the 32nd ACM International Conference on Advances in Geographic Information Systems, 715--716

    2024

  33. [41]

    and Yu, W

    Zhang, Y., He, Z., Li, J., Lin, J., Guan, Q. and Yu, W. (2024 a ) Mapgpt: an autonomous framework for mapping by integrating large language model and cartographic tools. Cartography and Geographic Information Science, 51, 717--743

    2024

  34. [42]

    and Yu, W

    Zhang, Y., Wang, Z., He, Z., Li, J., Mai, G., Lin, J., Wei, C. and Yu, W. (2024 b ) Bb-geogpt: A framework for learning a large language model for geographic information science. Information Processing & Management, 61, 103808

    2024

  35. [43]

    and Yu, W

    Zhang, Y., Wei, C., He, Z. and Yu, W. (2024 c ) Geogpt: An assistant for understanding and processing geospatial tasks. International Journal of Applied Earth Observation and Geoinformation, 131, 103976

    2024

  36. [44]

    , " * write output.state after.block = add.period write newline

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type url volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.senten...

  37. [45]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  38. [46]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.