Chain-of-Programming (CoP) : Empowering Large Language Models for Geospatial Code Generation

Anqi Zhao; Haoyue Jiao; Huayi Wu; Jianxun Wang; Jianyuan Liang; Shuyang Hou; Xiaopu Zhang; Zhangxiao Shen

REVIEW 4 major objections 5 minor 1 cited by

Chain-of-Programming (CoP) : Empowering Large Language Models for Geospatial Code Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A five-step prompt chain cuts LLM geospatial code hallucinations.

desk verdict A well-built geospatial code generation framework whose broad evaluation is undercut by unreleased artifacts and a headline comparison that the paper's own debug-loop data complicate. read the letter →

arxiv 2411.10753 v1 pith:IPEMTQI5 submitted 2024-11-16 cs.SE cs.AIcs.CL

Shuyang Hou , Haoyue Jiao , Zhangxiao Shen , Jianyuan Liang , Anqi Zhao , Xiaopu Zhang , Jianxun Wang , Huayi Wu This is my paper · ORCID

classification cs.SEcs.AIcs.CL

keywords geospatialcodegenerationlargelanguagemodelschain-of-programminghallucinationretrieval-augmentedpromptengineeringknowledgebasesGoogleEarthEngine

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a prompt-only framework called Chain of Programming (CoP) can substantially reduce code hallucination in geospatial code generation, where language models emit plausible-looking but non-executable or wrong code. CoP splits generation into five stages—requirement analysis, algorithm design, code implementation, debugging, and annotation—linked by a shared information pool, retrieval from curated geospatial knowledge bases, and user feedback, with no model fine-tuning. On 500 geospatial code-generation tasks across 17 models, the authors report improvements of 3.0 to 48.8 percentage points across matchability, executability, accuracy, and readability; for GPT-4, matchability rises from 62.6 to 97.2 and accuracy from 42.6 to 86.8. If the results hold, the practical message is that a large share of specialized code-generation quality can come from engineering the generation process and retrieving domain knowledge rather than retraining models.

What carries the argument

The load-bearing machinery is the five-stage CoP pipeline centered on a shared information pool. This pool is an explicit short-term memory, cleared after every task, that stores standardized JSON artifacts—the requirement document, the algorithmic workflow, and the generated code—so each stage reads the previous stage's output deterministically instead of relying on the model's stochastic recall. Around the pool sit three retrieval knowledge bases (platform or toolkit facts, function syntax, built-in datasets) and a feedback loop that feeds console error messages and correctness judgments back into the debugging stage. This design is what separates CoP from Chain-of-Thought and agent-based strategies: generation is anchored to stored artifacts and retrieved platform syntax rather than to whatever the model happens to know.

What would settle it

Run CoP on a fresh set of geospatial tasks that were not part of GeoCode-Eval, using knowledge bases built without access to those tasks, and compare matchability, executability, and accuracy against the reported numbers; if CoP's gains shrink toward the zero-shot baseline, the claim that the framework itself drives the improvement rather than task-specific knowledge bases is falsified.

Watch

Extended reading notes

Core claim

The central claim is that the dominant failure modes of large language models in geospatial code generation—unclear requirements, missing platform-specific syntax, wrong dataset identifiers, and unexecutable output—can be fixed by restructuring the workflow instead of retraining. CoP first runs a requirement-completeness check against eight elements (platform, language, objective, spatial extent, temporal extent, data source, method, and output), then decomposes the task into a JSON algorithm workflow, generates code while consulting a function-syntax knowledge base of 8,729 records and a built-in dataset knowledge base of 2,732 records, iterates debugging using user-supplied execution results up to a default of three rounds, and ends with code annotation. The paper reports that this pipeline improves all four evaluation metrics for all 17 tested models: for GPT-4, executability rises from 52.8 to 92.4 and accuracy from 42.6 to 86.8, while a small code model such as CodeGemma-7B improves accuracy from 12.0 to 49.6. Ablations show the largest gains come from combining all three supporting mechanisms, and debugging iterations show diminishing returns after the first pass.

Load-bearing premise

The results stand or fall on whether the hand-built geospatial knowledge bases and the expert ratings used by GeoCode-Eval measure real code quality honestly, compiled independently of the test tasks; the paper does not release the benchmark or knowledge bases, does not identify the expert raters, and tests no unseen tasks.

Editorial extensions

If this is right

CoP offers a low-cost path to better geospatial code: the same commercial or open-source LLM can be upgraded by prompt structure, retrieval, and feedback without fine-tuning or new training data.
The reported gains hold across model scale and type, from large commercial models to 6B open code models, so the method transfers rather than helping only the strongest models.
On the paper's benchmark, CoP outperforms other prompt, reasoning, and multi-agent optimization strategies, and the full combination of shared information pool, retrieval, and feedback is needed for the best results.
The debugging analysis implies that one maintenance round captures most of the benefit, so three rounds is a cost-effective default in practice.
The annotation stage measurably improves readability, a dimension where the other tested optimization strategies often hurt.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the same five-stage structure with a shared information pool and retrieved syntax knowledge should transfer to other vertical domains with closed, syntax-heavy APIs—for instance climate model configuration or bioinformatics pipelines—wherever curated function and dataset registries exist.
Editorial inference: the ablation pattern suggests an automated feedback model trained on console logs could replace the human expert in the debugging loop, making CoP scalable without fine-tuning.
Editorial inference: because smaller models gain more in relative terms, CoP may let compact open-source models approach the geospatial code quality of large closed models, which would lower the cost of deploying such assistants.
Editorial inference: releasing the knowledge bases and benchmark alongside an independent held-out test set would let other groups measure how much of CoP's gain transfers to unseen tasks and platforms.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The CoP paper is a solid systems contribution: it packages known techniques (retrieval, staged prompts, iterative debugging, shared memory) into a concrete geospatial code generation workflow and evaluates it on 17 base models against 15 existing strategies. The framework description is clear, the ablations are systematic, and the authors are upfront that the debugging stage is a hyperparameter. That is real engineering value.

The soft spots are concentrated in the evidence. The evaluation uses GeoCode-Eval, a benchmark the same group built, plus knowledge bases they compiled, and none of it is released—data availability is “on request.” Expert ratings have no stated number of annotators (beyond “five experts” for readability), no inter-rater reliability, and no error bars or significance tests. For an engineering paper claiming 3–48 point gains, that is a thin pedestal.

The bigger issue is that the headline comparison conflates the chain with the debugging loop. EQ1/EQ2 compare CoP with up to three oracle-backed debugging iterations against zero-shot and other strategies that get one shot. Table 8 shows the debugging loop alone is worth a lot: for GPT-4, CoP accuracy goes 50.8→86.8 from Debugging@0 to @3. The stress-test note claims there is no retry-with-same-oracle control, but Table 8 actually includes zero-shot-with-debugging rows. That control tells a more nuanced story: CoP@3 beats zero-shot@3 by 19.5 points of executability and 27.6 of accuracy in GPT-4, so the chain components do add value on top of retries. But the paper never makes that comparison explicit, and at Debugging@0, CoP actually loses to zero-shot@3 on all three models reported. That should have been discussed.

What I would ask for in revision: release the benchmark, KBs, prompts, and code; run the comparison with matched debugging budgets as the primary result; add simple significance checks; and clarify who the “experts” are. The contribution is worth a serious referee—it is a practical problem and the system is well-built—but the current evidence does not support the marketing-level claims.

Referee Report

4 major / 5 minor

Summary. The paper proposes Chain-of-Programming (CoP), a prompt-only framework that decomposes geospatial code generation into five stages—requirement analysis, algorithm design, code implementation, code debugging, and code annotation—supported by a shared information pool, retrieval from hand-built knowledge bases, and user/expert feedback. The authors evaluate CoP against zero-shot and a range of established prompting and agent strategies across 17 LLMs using four expert-rated metrics (matchability, executability, accuracy, readability) on a new benchmark, GeoCode-Eval, and report large improvements (3.0%–48.8%). Two case studies (building visualization and fire data analysis) illustrate the workflow.

Significance. The intended contribution is valuable: a systematic, fine-tuning-free approach to a specialized code generation domain, together with domain knowledge bases and a benchmark. If the reported findings are robust and the assets are released, this would be a useful step for geospatial code generation and a template for other vertical domains. The paper includes extensive empirical work with many models and strategies, and the staged design with a shared information pool is a reasonable contribution. However, the current evaluation does not support the headline claims because the CoP pipeline is compared to baselines without an equivalent oracle-driven debugging loop, and the benchmark and knowledge bases are not independently validated or released.

major comments (4)

[Sec. 4.3.4, Table 8; Sec. 4.3.1, Table 5] The headline comparison between CoP and zero-shot (and CoP vs. other strategies in Table 6) conflates the chain components with the oracle-guided debugging loop. Table 8 shows that for GPT-4, CoP accuracy rises from 50.8 at Debugging@0 to 86.8 at Debugging@3, and executability from 66.2 to 92.4; the corresponding zero-shot scores are 42.6 and 52.8. Thus the chain alone (Debugging@0) contributes only +8.2 accuracy, while the debugging loop contributes +36.0. Because the paper does not provide an equivalent retry-with-feedback control for zero-shot generation or for the comparison strategies (e.g., Self-Debugging, Reflexion, RAP in Table 6), the reported gains of CoP over these baselines cannot be attributed to CoP's novel mechanisms. Please either (a) ablate the debugging loop for all baselines and report CoP with and without it separately, or (b) explicitly acknowledge and quantify that the majority of the improvement stems from the generic iterative correction stage. This is load-bearing for the paper's central claim.
[Sec. 3.2.2 and Sec. 4.3] The evaluation relies entirely on GeoCode-Eval, a benchmark constructed by the same authors, and on expert ratings for all four metrics, yet neither the benchmark, the knowledge bases, nor the evaluation protocol is released. The paper states (Data availability, end of Section 6) that data are available on request, but provides no link and no detail on which items can be shared. Moreover, the expert-assessment process is under-specified: the number of experts, their domain expertise, the annotation instructions, and inter-rater reliability are not reported, so the reader cannot judge the reliability of the matchability, accuracy, and readability scores. Without third-party or independent validation, and without evidence that the 500 test tasks are disjoint from the content of the knowledge bases, the current experiments do not establish that the gains transfer to unseen geospatial tasks.
[Sec. 4.2, Table 6] The comparison against other optimization strategies is not apples-to-apples. Many baselines in Table 6 (e.g., Self-Debugging, Reflexion, RAP, AgentCoder) are iterative methods that can use feedback, but the paper does not state how many iterations they were allowed or whether they received the same oracle signal (executability/accuracy Y/N plus console errors) as CoP. If they were run for a single pass while CoP ran with up to three debugging iterations, the comparison understates the baselines. Please specify the exact protocol for each baseline, including iteration counts and feedback signals, and add a condition in which the strongest baselines receive the same number of correction chances.
[Sec. 4.3.4, Table 8] The paper's abstract and Section 4.3.1 present CoP as a full pipeline with the default debugging limit of three iterations, but Table 8 shows that the Debugging@0 condition (CoP without the maintenance loop) yields much smaller improvements over zero-shot. The reader is not told whether the EQ4 'Debugging@0' scores are the same as the 'CoP without feedback' row in the EQ3 ablation (Table 7), which would help disentangle the contribution of the debugging stage. Please clarify the relationship between these two ablations and, in all summary statements, report the chain-only improvement alongside the full-pipeline improvement.

minor comments (5)

[Section 6] The conclusion uses 'Chain-of-Processing' while the title and elsewhere use 'Chain-of-Programming'; please unify the terminology.
[Section 3.1] The text says 'The prompt template and standardized information for code implementation are illustrated in Figure 6,' but Figure 6 is described as showing the code debugging and commenting prompts; please fix the figure cross-reference.
[Table 5] Table 5 is hard to read because the columns for zero-shot and CoP are not clearly separated and several numeric values appear misaligned; please reformat the table so each metric clearly distinguishes the two conditions.
[Section 4.3.3, Table 7] The row labeled '× × ×' in Table 7 is not explicitly defined; please state clearly that this is CoP without any of the three mechanisms, not zero-shot, to avoid confusion with Table 5.
[Abstract; Sec. 4.3.1] The improvement range '3.0% to 48.8%' is reported without specifying which model and metric each endpoint corresponds to; please give the exact source of these bounds.

Circularity Check

1 steps flagged · score 4.0 of 10

The headline Accuracy/Executability gains are partially reduced to the debugging loop's use of the evaluation oracle, while the chain components themselves are not definitionally circular.

other [Section 3.1 (code debugging) and Section 4.3.4 (EQ4)]
"if it is executable but the results are incorrect, feedback on the unexpected output is provided through console interaction, and the debugging process resumes for correction ... If the code fails to execute correctly or the results do not meet expert expectations, error messages from the console or outputs that do not match expectations can be fed into the CoP process for further correction."

CoP's Accuracy and Executability are measured after the debugging loop has consumed the same correctness signal used to define those metrics. The loop is told whether results 'do not meet expert expectations' and receives 'outputs that do not match expectations' as input; the corrected code is then scored on alignment with expected outcomes. Zero-shot and most EQ2 baselines generate once without this oracle feedback, so the reported gains (e.g., GPT-4 Accuracy 50.8 at Debugging@0 to 86.8 at Debugging@3) attribute to the CoP chain a component that is mechanically due to iterative fitting against the evaluation oracle. This is a partial reduction of a headline metric to an input signal, not an independent measurement of the chain components.

full rationale

This is an empirical systems paper, so there is no mathematical derivation whose conclusion is identical to its premises. The CoP framework is not defined in terms of its reported scores, and Debugging@0 results plus EQ3 ablations provide some independent evidence for the chain components. However, the evaluation protocol for the headline Accuracy and Executability claims is partially circular: the debugging stage feeds correctness feedback derived from the same expected-outcome labels that define Accuracy, and the final scores are computed on the corrected outputs. The paper does not run the comparison strategies with an equivalent retry-with-oracle control, so the magnitude of CoP's improvement over zero-shot and other strategies is not cleanly attributable to the shared information pool, knowledge retrieval, or staged prompts. Additionally, the evaluation benchmark GeoCode-Eval is self-created (Hou et al., 2024b), but this is an independence limitation rather than a definitional reduction. Overall, the central claim retains independent content, so the circularity is partial rather than total.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The framework introduces no new physical or mathematical entities. The free parameter is the debugging iteration limit. The main assumptions are about the validity and independence of the evaluation artifacts and the stability of LLM behavior.

free parameters (1)

maximum debugging iterations = 3 (default)
Chosen by hand as a hyperparameter; EQ4 shows performance varies with this value, indicating it is a free parameter affecting the central results.

assumptions (3)

domain assumption The GeoCode-Eval benchmark and expert ratings provide a valid and unbiased measure of geospatial code generation quality.
The evaluation depends on this assumption; the benchmark was created by the authors and expert evaluators are not described as independent (Section 3.2.2, Section 4.3).
domain assumption The manually compiled knowledge bases (platform, function syntax, built-in datasets) are accurate, comprehensive, and built independently of the test tasks.
CoP's retrieval mechanism uses these knowledge bases to improve code generation; if they overlap with benchmark questions, the measured gains may not generalize (Section 3.2.1).
domain assumption LLM API behavior is stable across evaluation runs and reflects typical model performance.
The paper does not report multiple runs or variance, so it implicitly assumes deterministic or stable model outputs (Section 4.3).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Chain-of-Programming (CoP) : Empowering Large Language Models for Geospatial Code Generation." pith.science (2026). https://pith.science/paper/IPEMTQI5

@misc{pith2026241110753,
  author       = {Pith},
  title        = {Pith review of: Chain-of-Programming (CoP) : Empowering Large Language Models for Geospatial Code Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IPEMTQI5}},
  note         = {Machine review of arXiv:2411.10753}
}

read the original abstract

With the rapid growth of interdisciplinary demands for geospatial modeling and the rise of large language models (LLMs), geospatial code generation technology has seen significant advancements. However, existing LLMs often face challenges in the geospatial code generation process due to incomplete or unclear user requirements and insufficient knowledge of specific platform syntax rules, leading to the generation of non-executable code, a phenomenon known as "code hallucination." To address this issue, this paper proposes a Chain of Programming (CoP) framework, which decomposes the code generation process into five steps: requirement analysis, algorithm design, code implementation, code debugging, and code annotation. The framework incorporates a shared information pool, knowledge base retrieval, and user feedback mechanisms, forming an end-to-end code generation flow from requirements to code without the need for model fine-tuning. Based on a geospatial problem classification framework and evaluation benchmarks, the CoP strategy significantly improves the logical clarity, syntactical correctness, and executability of the generated code, with improvements ranging from 3.0% to 48.8%. Comparative and ablation experiments further validate the superiority of the CoP strategy over other optimization approaches and confirm the rationality and necessity of its key components. Through case studies on building data visualization and fire data analysis, this paper demonstrates the application and effectiveness of CoP in various geospatial scenarios. The CoP framework offers a systematic, step-by-step approach to LLM-based geospatial code generation tasks, significantly enhancing code generation performance in geospatial tasks and providing valuable insights for code generation in other vertical domains.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

AutoGEEval: A Multimodal and Automated Framework for Geospatial Code Generation on GEE with Large Language Models
cs.SE 2025-05 conditional novelty 6.0 of 10

An automated, execution-based benchmark of 1,325 Google Earth Engine unit tests shows 18 LLMs scoring between 31.40% and 71.55% pass@1, with parameter-knowledge errors the dominant failure mode.

Reference graph

Works this paper leans on

6 extracted references · 5 canonical work pages · cited by 1 Pith paper

[1]

geospatial code(Bebortta et al., 2020; Hou et al., 2024c)

Introduction The rapid advancements in remote sensing and sensor network technologies have significantly enhanced geospatial data acquisition and processing, resulting in a dramatic increase in data volume(Breunig et al., 2020; Zhang et al., 2022a). Concurrently, the growing importance of spatiotemporal data in fields such as ecology, transportation, poli...

work page 2020
[2]

information silos(Ning et al., 2023; Pan et al., 2024)

Related Work 2.1. Code Generation with Large LLMs Code generation tasks focus on converting natural language inputs into source code (NL2Code)(Dehaerne et al., 2022; Jiang et al., 2024). Early approaches in this field relied on heuristic rules(Nymeyer and Katoen, 1997), expert systems(Depradine, 2003), or reinforcement learning methods(Le et al., 2022)—su...

work page 2022
[3]

Chain of Programming 3.1. Overall Framework The overall framework of CoP consists of five interconnected chain-like processes: requirement analysis, algorithm design, code implementation, code debugging, and code annotation, resembling the structured operational flow of the waterfall model. Each stage facilitates the transfer and sharing of structured inf...

work page
[4]

no maintenance

Evaluation The evaluation aims to scientifically quantify the following issues across various mainstream LLMs: EQ1：The performance change of the CoP strategy compared to zero-shot performance. EQ2：The performance change of the CoP strategy compared to other optimization strategies. EQ3：The necessity of the three key mechanisms in the CoP strategy: shared ...

work page 2023
[5]

User Requirement Supplementation

Case Study To comprehensively demonstrate the effectiveness of the CoP strategy in generating geospatial code, this study selects two representative geospatial code generation scenarios for case analysis. These two scenarios are representative across multiple dimensions: the first case study region is located in China, while the second is in an internatio...

work page 2022
[6]

coding hallucination

Conclusion This paper introduces the Chain-of-Processing (CoP) programming framework, which builds an end-to-end pipeline from requirement analysis, algorithm design, code implementation, testing, and maintenance, aimed at overcoming the “coding hallucination” problem currently faced by LLMs in geospatial code generation tasks. The framework enhances the ...

arXiv 2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

geospatial code(Bebortta et al., 2020; Hou et al., 2024c)

Introduction The rapid advancements in remote sensing and sensor network technologies have significantly enhanced geospatial data acquisition and processing, resulting in a dramatic increase in data volume(Breunig et al., 2020; Zhang et al., 2022a). Concurrently, the growing importance of spatiotemporal data in fields such as ecology, transportation, poli...

work page 2020

[2] [2]

information silos(Ning et al., 2023; Pan et al., 2024)

Related Work 2.1. Code Generation with Large LLMs Code generation tasks focus on converting natural language inputs into source code (NL2Code)(Dehaerne et al., 2022; Jiang et al., 2024). Early approaches in this field relied on heuristic rules(Nymeyer and Katoen, 1997), expert systems(Depradine, 2003), or reinforcement learning methods(Le et al., 2022)—su...

work page 2022

[3] [3]

Chain of Programming 3.1. Overall Framework The overall framework of CoP consists of five interconnected chain-like processes: requirement analysis, algorithm design, code implementation, code debugging, and code annotation, resembling the structured operational flow of the waterfall model. Each stage facilitates the transfer and sharing of structured inf...

work page

[4] [4]

no maintenance

Evaluation The evaluation aims to scientifically quantify the following issues across various mainstream LLMs: EQ1：The performance change of the CoP strategy compared to zero-shot performance. EQ2：The performance change of the CoP strategy compared to other optimization strategies. EQ3：The necessity of the three key mechanisms in the CoP strategy: shared ...

work page 2023

[5] [5]

User Requirement Supplementation

Case Study To comprehensively demonstrate the effectiveness of the CoP strategy in generating geospatial code, this study selects two representative geospatial code generation scenarios for case analysis. These two scenarios are representative across multiple dimensions: the first case study region is located in China, while the second is in an internatio...

work page 2022

[6] [6]

coding hallucination

Conclusion This paper introduces the Chain-of-Processing (CoP) programming framework, which builds an end-to-end pipeline from requirement analysis, algorithm design, code implementation, testing, and maintenance, aimed at overcoming the “coding hallucination” problem currently faced by LLMs in geospatial code generation tasks. The framework enhances the ...

arXiv 2024