{"id":"6783daad-44c5-49d3-9a5f-cc715823b8f4","arxiv_id":"2505.12900","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An automated, execution-based benchmark of 1,325 Google Earth Engine unit tests shows 18 LLMs scoring between 31.40% and 71.55% pass@1, with parameter-knowledge errors the dominant failure mode.","lead":"AutoGEEval is an automated test suite that checks whether AI models can write working Google Earth Engine code: 1,325 unit tests across 26 geospatial data types, with pass or fail decided by actually running the code. The paper ranks 18 language models on this suite and finds the biggest failure is missing geospatial knowledge, such as dataset paths and band names, rather than broken syntax.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground truth is Qwen2.5-Max output with no demonstrated independence; the benchmark's central claim depends on expert revision actually catching subtle reference errors.","rationale":"The reader's weakest assumption is exactly ground-truth quality and independence from Qwen conventions. The paper is otherwise a substantial, execution-based benchmark with plausible pass@n results and internal consistency (error-type patterns, diminishing returns). The central claim is the validity of the benchmark and the measured rankings. The most load-bearing risk is that the 1325 expected answers inherit Qwen2.5-Max-specific choices that the judge enforces, so model rankings reflect fidelity to one LLM's style rather than to actual GEE semantics. The concrete test is direct: expose the references and revision artifacts and independently audit a sample. This is feasible because the authors have the data and the GitHub repository; it does not require re-running all experiments. I agree with the reader that this warrants a CONDITIONAL rather than unconditional ACCEPT, primarily on reproducibility and ground-truth transparency. The Table 9 inconsistency is also mentioned, but the ground-truth assumption is the more load-bearing issue because it threatens all 18-model comparisons, not just the efficiency ranking table.","tokens_in":23548,"tokens_out":1931,"duration_ms":16745,"concrete_test":"Release the expert-revision log (or a random audit of 100 cases) showing original versus revised Reference_code and Expected_answer for every test case, together with the raw judge outputs. Then independently re-run the 100 audited cases with a second, independent expert panel; if any revised reference still contains an execution-valid alternative output judged wrong, or if the second panel changes more than 5 of 100 references, the reported rankings are not robust to ground-truth bias.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The benchmark's ground truth is derived from LLM-written reference code. Section 3.3: reference code is generated by Qwen2.5-Max, then executed and revised by five experts in about 50 person-hours for 1325 cases (~2.3 min/case). The paper reports no per-case revision log, no inter-annotator agreement, no measurement of residual reference errors, and no audit showing which cases were changed. If any reference contains a subtle error or encodes a Qwen-specific convention (default arguments, band ordering, string formatting, geographic projection details), the judge will mark valid alternative implementations wrong. Because pass@1 spans 31.40% to 71.55%, a systematic bias in a small fraction of references could shift rankings. Table 3 compounds this: large-image comparisons use center sampling with tolerance 0.001, so correctness away from the center is unverified and approximate value matching can reject functionally correct outputs. The paper's own limitations (7.2) acknowledge only coverage and metric scope, not possible reference bias. Thus the benchmark's validity rests on an unverified independence assumption: that expert review was sufficient to purge LLM-specific errors from all 1325 expected answers.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AutoGEEval, an automated, unit-level evaluation framework for code generation on the Google Earth Engine (GEE) Python API. It constructs AutoGEEval-Bench, a set of 1,325 test cases covering 26 GEE output types, where each test case contains a function header, a reference code snippet, parameters, output type, output path, and an expected answer. A submission program prompts target LLMs to generate function bodies, executes them, and saves outputs; a judge program compares outputs with expected answers using type-specific matching strategies. The authors evaluate 18 LLMs (general, reasoning, code-specific, and geospatial) and report pass@1/pass@3/pass@5, coefficient of variation, stability-adjusted accuracy, token consumption, inference time, code lines, efficiency ratios, and error type distributions. The main claims are that AutoGEEval is the first multimodal unit-level automated evaluation framework for GEE code generation and that the results characterize model capabilities, with parameter errors being the dominant failure mode.","tokens_in":23587,"tokens_out":10444,"duration_ms":85746,"significance":"If the benchmark and judge are valid, AutoGEEval-Bench and the open-source framework constitute a valuable community resource for reproducible evaluation of geospatial code generation. The broad model comparison (18 models), the fine-grained error type analysis, and the inclusion of resource and efficiency metrics are useful contributions. The paper also ships the benchmark and code publicly (GitHub link in the data availability statement). However, the validity of the ground truth and the correctness of the efficiency tables are load-bearing for these contributions.","major_comments":[{"comment":"The In.T-E and Co.L-E columns in Table 9 are not computed according to Eqs. (10) and (12). For example, GPT-4o has Co.L-E = 7.77, which equals its Co.L value in Table 8 rather than pass@5/Co.L = 65.36/7.77 (approx. 8.41); DeepSeek-Coder-V2 has In.T-E = 8.39, which equals its In.T value rather than 40.75/8.39 (approx. 4.86). The error affects all locally deployed models for In.T-E and all models for Co.L-E, and consequently invalidates Co_Rank, E_Rank, and Total_Rank in Table 10. The authors must recompute these columns and the resulting rankings.","section":"Table 9 / Eqs. (10) and (12)"},{"comment":"The ground truth (Expected_answer) is produced by executing reference code generated by Qwen2.5-Max, with expert revision reported only as 50 person-hours for 1,325 cases (about 2.3 minutes per case). The paper provides no per-case revision log, no inter-annotator agreement, and no audit of residual reference errors. Because six of the 18 evaluated models (Qwen-2.5-3B/7B/32B and Qwen2.5-Coder-3B/7B/32B) belong to the same model family as the oracle generator, any Qwen-specific conventions in the references (e.g., default arguments, band ordering, string formatting) could systematically disadvantage other models and bias the rankings. Please provide evidence that expert revision removed such conventions, e.g., a sample audit, a list of revised cases, or an independent re-verification subset.","section":"Section 3.3 / Table 5 / Eq. (6)"},{"comment":"For ee.Image and ee.ImageCollection, the judge uses center sampling with a tolerance of 0.001 for large images. This means that correctness of the entire image is gated by a small central region. The paper does not report how many of the 224 ee.Image test cases fall into the 'large image' category, nor does it justify that center sampling is sufficient. If a functionally correct implementation differs from the reference only outside the sampled center (or within tolerance but in a spatially varying way), it will be marked wrong. Please quantify the impact and provide evidence that this approximation does not change the reported rankings.","section":"Section 4.2 / Table 3"},{"comment":"The definition of pass@n is ambiguous and internally inconsistent. The text says N is the total number of generated samples and C_n is the number of incorrect samples, which would make pass@n independent of n; however, the reported pass@3 and pass@5 values are larger than pass@1, implying that C_n actually counts test cases for which all n attempts fail. Please define C_n and N precisely, state how the n samples per test case are aggregated, and explain whether this is the standard pass@k estimator or a different plug-in estimator.","section":"Section 5.3.1 / Eq. (7)"}],"minor_comments":[{"comment":"The task definition states that code is generated 'within the Earth Engine JavaScript environment,' but the framework is implemented with the GEE Python API (Section 1). Please correct this inconsistency.","section":"Section 3.1 / Eq. (2)"},{"comment":"The data type 'ee.Pixeltype' should be 'ee.PixelType' to match the GEE naming convention.","section":"Table 3"},{"comment":"QwQ-32B is assigned T_Rank 18 even though its Tok.-E is N/A. If a metric is unavailable, the ranking should exclude the model from that dimension rather than ranking it last, or the paper should justify the choice.","section":"Table 10"},{"comment":"The reported average improvements (12.88% for pass@3, 3.81% for pass@5) and the ranges do not match Table 6. For example, the largest pass@3 improvement in Table 6 is 12.45 percentage points (DeepSeek-R1), not 21.37%, and the computed average of the 18 pass@3 improvements is approximately 6.6 percentage points. Please recompute these summary statistics.","section":"Section 6.1 / text near Figure 9"},{"comment":"The related-work discussion acknowledges GeoSpatial-Code-LLMs but does not explicitly state how AutoGEEval's automation differs from it beyond sample size and data-type coverage; please make the comparison concrete.","section":"Section 2.3"}],"recommendation":"major_revision","confidential_remarks":"The paper makes a solid empirical contribution, but the ground-truth independence issue and the Table 9 computation errors are load-bearing and need to be addressed before publication. I recommend major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about AutoGEEval. First, the benchmark and judge are real infrastructure: 1,325 unit-level test cases covering 26 GEE output types, with execution-based comparison instead of manual scoring. That is a genuine step beyond GeoCode-Bench, GeoCode-Eval, and the small GeoSpatial-Code-LLMs dataset, and it makes reproducible model comparison on GEE plausible. Second, the paper's efficiency-ranking table (Table 9) has a clear data error: the Co.L-E column repeats the Co.L column from Table 8 instead of computing pass@5 / Co.L as Eq. (12) defines. For GPT-4o, Co.L-E is listed as 7.77, which is the code line count, not 65.36/7.77 approx 8.41. This propagates into E_Rank and Total_Rank. The other efficiency columns look correctly computed, so this looks like a copy-paste mistake rather than systematic fabrication, but it needs to be fixed and the rankings re-run.\n\nThe accuracy results themselves are plausible and internally consistent: pass@1 from 31.4% to 71.55%, diminishing returns from pass@3 to pass@5, and parameter errors as the dominant error type. The 18-model comparison is useful, and the category-level observations (reasoning models cost more tokens, GeoCode-GPT improves only modestly on Code-Llama) are sensible.\n\nThe bigger soft spot is the ground truth. Reference code is generated by Qwen2.5-Max, then executed and revised by five experts in roughly 50 person-hours for 1,325 cases, about 2.3 minutes per case. The paper gives no per-case revision log, no inter-annotator agreement, and no audit of which references were changed. Since six of the eighteen evaluated models are Qwen-family, a systematic bias in the oracle would not be neutral. This is not a fatal flaw, because many of the 1,325 tasks are simple function calls where reference generation is likely reliable, but the paper needs to show the expert review was effective. At minimum, release the revised test cases, report the fraction changed, and provide a spot-check protocol.\n\nTwo smaller issues: Table 3 uses center sampling with a 0.001 tolerance for large images, which can reject correct outputs that differ away from the center, and the rankings have no error bars or significance tests, so adjacent ranks may be within noise. Also, Section 7.2 calls resource usage future work even though Sections 5.3.2 through 6.3 already report it; that is a wording slip, not a deep issue.\n\nWho this is for: researchers working on geospatial code generation or LLM evaluation infrastructure. It deserves a serious referee, and my recommendation is conditional acceptance after the Table 9 fix and a concrete plan for auditing the reference answers.","headline":"AutoGEEval ships a genuinely reusable execution-based benchmark and judge for GEE code generation, but the efficiency table has a copy-paste error and the Qwen-written oracle needs an audit before the rankings are trusted.","tokens_in":24373,"tokens_out":3499,"would_cite":true,"duration_ms":32732,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AutoGEEval is an automated benchmark for measuring whether large language models can write Google Earth Engine code, tested on 1,325 API functions across 18 models.","keywords":["geospatial code generation","Google Earth Engine","large language models","automated evaluation","unit test benchmark","LLM evaluation","remote sensing","code generation"],"falsifier":"Independently rewrite the expected answers for a random sample of AutoGEEval test cases from a different codebase or by GEE experts who have not seen the stored references, then rerun the 18 models through the Judge Program. If model pass rates shift materially, or if human experts disagree with the judge on a meaningful share of outputs, the framework's verdicts are not neutral.","tokens_in":23144,"feed_emoji":"🛰️","tokens_out":9434,"duration_ms":90890,"temperature":0.7,"pith_summary":"This paper claims that geospatial code generation lacks a reproducible, automated measuring stick, and that AutoGEEval supplies one. It introduces a benchmark of 1,325 unit-level test cases, one per valid function in the Google Earth Engine client API documentation, spanning 26 output data types from images and geometries to lists and strings. The framework runs a model's generated function body with fixed parameters, executes it against the GEE Python API, and a Judge Program compares the saved output with a stored expected answer. Applied to 18 large language models, it reports pass@1 scores ranging from 31.40% to 71.55%, with parameter errors as the dominant failure mode. The paper concludes that current models' core weakness is domain-specific GEE knowledge, not general programming syntax.","feed_headline":"New benchmark ranks 18 AI models at writing Earth Engine code","feed_subtitle":"AutoGEEval automates 1,325 unit tests across the Google Earth Engine API; parameter mistakes, not syntax, sink most models.","key_machinery":"The load-bearing mechanism is the execution-and-compare loop. A test case stores the official documentation's function header plus a hidden reference implementation; the Submission Program prompts the target model to write only the function body, injects concrete parameters, runs the code on the GEE Python API, and writes the result to a designated path. The Judge Program then dispatches on the declared output type and compares the produced value to the expected answer using per-representation matching rules, with pass@n counting how often a correct output appears in n attempts. The framework also logs tokens, inference time, code lines, and runtime error categories, so the same loop produces cost and stability metrics rather than a single accuracy number.","core_discovery":"AutoGEEval is, the authors argue, the first multimodal unit-level automated evaluation framework for GEE code generation. Each test case is a six-tuple: function header, reference code, parameter list, output type, output path, and expected answer. During testing the reference code is hidden, parameters are injected, and the Judge Program maps the 26 declared GEE types onto runtime value representations—arrays, dictionaries, GeoJSON, lists, floats, and strings—applying a matching strategy for each; large images are validated by center sampling with a tolerance of 0.001. On this benchmark, pass@1 ranges from 31.40% to 71.55% and pass@5 reaches as high as 77.28%, while multiple generation attempts show sharply diminishing returns after pass@3. The paper reads the error logs as evidence that models have mature syntax skills but systematic gaps in GEE-specific parameter knowledge such as dataset identifiers, band names, and coordinate formats.","pith_inferences":["My inference: because the stored reference answers were drafted by one large language model and only human-reviewed, the benchmark may systematically favor outputs stylistically similar to that model's conventions, such as argument ordering, default values, or string formatting; an independent set of expert-written references could reorder the rankings.","My inference: the judge's center-sampling rule for large images means a model that computes the correct geospatial result but differs slightly at the sampled center could be marked wrong; a human re-scoring study of image-type outputs would quantify that risk.","My inference: the same framework could be extended to multi-function composition and cross-API tasks, and to retrieval-augmented variants, providing a controlled test of whether retrieval closes the parameter-error gap; this is a natural next experiment the paper does not run."],"forward_implications":["If AutoGEEval is right, future geospatial model evaluation can shift from expert manual scoring to an automated loop, making results reproducible across labs and models.","The error-type logs imply that targeted injection of GEE domain knowledge, such as dataset names, band names, and argument formats, should yield larger accuracy gains than further general code training, because parameter errors dominate failures.","The cost metrics imply that reporting accuracy alone is misleading: reasoning-augmented models can score competitively on pass@n while consuming several times more tokens and latency, and multi-sample gains diminish after the third attempt.","The 18-model comparison implies that model rank is not determined by category or size: a large general model leads, a code-specialized model from the same family is last, and the fine-tuned geospatial model improves over its base model by a modest margin.","The one-function-per-case scope means the measured capability is API-call correctness, not multi-step analytical workflow construction."],"supporting_citations":[{"why":"Supplies the code-specialized model family whose evaluated variant appears in the comparison and ranks last.","marker":"[6]"},{"why":"Supplies the code-generation model family evaluated at three parameter sizes in the comparison.","marker":"[7]"},{"why":"Provides the base model for the fine-tuned geospatial model, enabling the fine-tuning benefit comparison.","marker":"[8]"},{"why":"Supplies the generic code-generation benchmark and pass@n evaluation convention that AutoGEEval adapts to geospatial tasks.","marker":"[10]"},{"why":"Provides the geospatial code generation model included as the task-specific category in the 18-model comparison.","marker":"[24]"},{"why":"Documents the earlier geospatial code benchmark that used manual expert scoring, the gap AutoGEEval targets.","marker":"[27]"},{"why":"Offers an earlier automated geospatial code evaluation dataset whose limited modality coverage and small sample motivate the new benchmark.","marker":"[28]"}],"fun_headline_variants":["AutoGEEval: first automated framework to test GEE code generation","18 AI models benchmarked on 1,325 Google Earth Engine coding tasks","Parameter errors, not syntax, trip up LLMs on GEE code, new study shows","Automated GEE benchmark: best LLM hits 71.55% pass@1","New AutoGEEval framework auto-evaluates LLM geospatial code"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the expected answers against which every model is judged are correct and unbiased; they were written by one large language model and then reviewed by human experts, so any error or stylistic convention baked into those reference answers would tilt the benchmark and the rankings.","fun_headline_variants_meta":{"raw":{"variants":["AutoGEEval: first automated framework to test GEE code generation","18 AI models benchmarked on 1,325 Google Earth Engine coding tasks","Parameter errors, not syntax, trip up LLMs on GEE code, new study shows","Automated GEE benchmark: best LLM hits 71.55% pass@1","New AutoGEEval framework auto-evaluates LLM geospatial code"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000723,"raw_usage":{"total_tokens":3261,"prompt_tokens":984,"completion_tokens":2277,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":600,"completion_tokens_details":{"reasoning_tokens":2171}},"tokens_in":600,"tokens_out":2277,"duration_ms":15493,"temperature":1.0,"reasoning_tokens":2171,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:26:12.052853+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Independently rewrite the expected answers for a random sample of AutoGEEval test cases from a different codebase or by GEE experts who have not seen the stored references, then rerun the 18 models through the Judge Program. If model pass rates shift materially, or if human experts disagree with the judge on a meaningful share of outputs, the framework's verdicts are not neutral.","supporting_citations":[{"cited_title":"GeoCode -GPT: A large language model for geospatial code generation","cited_arxiv_id":null,"evidence_quote":"Provides the geospatial code generation model included as the task-specific category in the 18-model comparison."}],"review_version":1}