Pith. sign in

REVIEW 3 major objections 5 minor 26 references

The paper claims that a two-stage LLM pipeline—semantic retrieval plus sandboxed Python code generation—can answer citizens' geospatial open-data questions with 98% analytical accuracy while rejecting unsupported questions.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A new LLM framework retrieves relevant geospatial open-data datasets, generates and executes analysis code in a sandbox, and rejects unanswerable queries, reaching 98% analytical correctness with GPT-4.1 on a 199-question benchmark.

T0 review reviewed 2026-08-03 challenge →

load-bearing objection A solid, well-described applied system with a useful released benchmark; the headline numbers are analysis-stage, LLM-judged, and the paper should be conditioned on showing the judge is trustworthy. the 3 major comments →

arxiv 2602.00012 v3 pith:NKSYYCT6 submitted 2025-11-30 cs.LG cs.AIcs.CYcs.IR

OGD4All: A Framework for Accessible Interaction with Geospatial Open Government Data Based on Large Language Models

Michael Siebenmann , Javier Argota S\'anchez-Vaquerizo , Stefan Arisona , Krystian Samp , Luis Gisler , Dirk Helbing This is my paper
classification cs.LG cs.AIcs.CYcs.IR
keywords open government datageospatial question answeringlarge language modelscode generation agentssemantic retrievalhallucination avoidancetrustworthy AIsandboxed execution
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

OGD4All is a proposal for making geospatial open government data usable by non-experts: an LLM first searches dataset metadata to find relevant public datasets, and if none exist it says so instead of inventing an answer. If datasets are found, a second agent repeatedly writes and runs Python code in a secure sandbox to produce a verifiable answer, map, plot, or table. On a 199-question benchmark built from a city's open data platform, the best evaluated model answers 98.2% of 169 answerable analysis questions correctly, retrieves relevant datasets with 94% recall, and nearly always rejects the 30 questions that have no data-backed answer. The paper's central argument is that replacing free-form text generation with executable, inspectable code is what keeps the system grounded and auditable. If this holds, it offers a reusable template for trustworthy citizen access to public data.

Core claim

The central claim is that a two-stage architecture—semantic dataset retrieval followed by an agentic Python code-generation loop in a sandboxed environment—can deliver near-expert answers to natural-language questions about geospatial open data. The best evaluated model reached 98.2% correctness on 169 answerable benchmark questions and 94% recall in retrieving the relevant datasets from a pool of 430, while correctly rejecting nearly all of 30 unanswerable questions. The paper reports that incorrect answers were traceable to code-level issues such as geometric approximations or dataset overlap, not to hallucinated data, and that every answer is backed by logged code and dataset links. The a

What carries the argument

The load-bearing mechanism is the two-stage retrieval-and-analysis loop. Retrieval uses dense vector embeddings of both metadata and the user's question, with an LLM allowed to reformulate the question into up to three subqueries and repeatedly call a nearest-neighbor search; if no relevant dataset is found, the interaction terminates—a built-in hallucination guard. Analysis then places each retrieved dataset in a persistent notebook environment where an LLM agent iteratively plans, generates Python code, executes it in a restricted sandbox, reads the output, and recovers from errors until it calls a final-answer tool. The key identity is that analysis is expressed as executable code, so the

Load-bearing premise

The 98% accuracy claim rests on a language-model judge whose own reliability is not quantitatively validated, so if that judge is systematically lenient or the hand-written ground-truth answers are not truly unique, the headline number is unsupported.

What would settle it

Have human annotators score a random subset of the 169 answerable benchmark answers using the same ground-truth scripts, then compare their verdicts with the model judge's verdicts; a significant disagreement on even a small sample would falsify the claimed 98.2% correctness.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Government data assistants can reject unanswerable questions by design, reducing hallucination risk below what prompt-level instruction alone achieves.
  • Citizens can inspect every intermediate step—reformulated queries, generated code, and dataset links—making errors localizable and correctable.
  • The large gap between frontier and open-weight models implies that sovereign, publicly hosted deployments need cheaper open models to close before they can match the best accuracy.
  • For cost-constrained deployments, a mid-tier model at 91.7% correctness and much lower API cost is a viable alternative to the frontier model.
  • Frontier models already saturate this benchmark, so measuring future progress will require harder questions, ambiguous wording, and multimodal output grading.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural extension beyond the paper is end-to-end evaluation: because retrieval and analysis are scored separately with ground-truth datasets supplied to the analysis stage, the real user-facing accuracy of the full pipeline remains unmeasured and could be lower.
  • The benchmark only grades textual answers on uniquely answerable questions, so the framework's behavior on ambiguous questions, open-ended prompts, and the quality of generated maps and plots is untested; citizen studies with real questions would reveal the practical ceiling.
  • The auditable code-generation design could transfer beyond government data—for instance to scientific open data or investigative journalism—wherever verifiability of a data-driven claim matters as much as its accuracy.
  • As open-weight models improve, the gap the paper documents suggests a path to fully sovereign deployments that keep the transparency benefits without depending on proprietary APIs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents OGD4All, a two-stage LLM framework for interacting with geospatial Open Government Data. Stage one retrieves relevant datasets from a metadata database via semantic search and agentic query reformulation, with an explicit rejection path for unanswerable questions. Stage two generates and executes Python code in a sandboxed notebook to answer the question and produce multimodal outputs. The authors contribute a 199-question benchmark (169 answerable, 30 unanswerable) built from 70 parameterized templates over City-of-Zurich data, and evaluate 11 LLMs on retrieval and analysis separately. They report that GPT-4.1 achieves 98.2% analytical correctness, 94% recall, and near-perfect answerability, while open-weight models lag. The paper also reports expert feedback from six municipal experts and emphasizes transparency, auditability, and reproducibility.

Significance. If the evaluation withstands scrutiny, the framework is a useful step toward trustworthy LLM-based access to open government data: it integrates a rejection mechanism for unsupported queries, makes generated code and traces auditable, and ships a reproducible benchmark with ground-truth answer scripts. The public repository, released traces, cost accounting, and parameterized benchmark are genuine strengths. However, the headline correctness number depends on an unvalidated GPT-4o judge, and the retrieval and analysis stages are never evaluated end-to-end. These issues are fixable but currently leave the central quantitative claims weaker than the abstract suggests.

major comments (3)
  1. [Section III-C] The central analytical-correctness claim (98.2% for GPT-4.1, Fig. 6) rests on an LLM-as-a-judge (GPT-4o). The paper only says that 'preliminary experiments' showed the judge is more robust than substring/fuzzy matching; it provides no human agreement study, no per-item judge decisions, no confusion analysis, and no breakdown by answer type. Because many of the 169 reference answers are numeric ('a number, name, or location'), even minor judge leniency on rounding, units, or decimal formatting could shift the headline by several points. The judge belongs to the same model family as the best-performing systems, which raises the risk of style-similarity inflation. Please report a human-validated calibration of the judge on a stratified sample of benchmark answers — including near-miss numeric cases — and publish judge prompts/outputs or an exact-match/tolerance check for numeric answers.
  2. [Section III-C / IV] Retrieval and analysis are evaluated separately; for the analysis runs, ground-truth relevant datasets are supplied, so the reported metrics are component-level, not end-to-end. The abstract's wording — 'OGD4All reaches 98% analytical correctness and 94% recall while reliably rejecting questions unsupported by available data' — can readily be read as a system-level claim. If the intended claim is per-component, please state this explicitly in the abstract and conclusion. If a user-facing system-level claim is intended, run the complete pipeline on the 199-question benchmark and report the combined success/rejection rate, which is currently missing.
  3. [Abstract / Section IV] The abstract promises 'statistical robustness tests,' but I could not find any in the results section: there are no confidence intervals, significance tests, or cross-run variability estimates, only point estimates and medians. Without this analysis, the claim that the framework 'shows reliability' is unsubstantiated. Either add the statistical analysis (and cite where it appears) or remove the phrase from the abstract.
minor comments (5)
  1. [Abstract] The headline percentages should be attributed to GPT-4.1 and labeled as stage-specific (retrieval vs. analysis). As written, the abstract could imply that the reported 98% and 94% apply to all 11 models or to the full pipeline.
  2. [Section IV-A] The claim that GPT-4o scores 5% higher in precision than GPT-4.1 is not supported by a numeric table. Please report exact precision/recall/answerability values for all models in a table or appendix.
  3. [Section IV-B] The statement that 'no incorrect claims were produced through LLM-generated, hallucinated data' is a strong negative claim. Clarify how this was audited (e.g., manual inspection of all error traces?) and report the number of traces examined.
  4. [Section IV-B] The example question ID 6 with a 'small bias due to dataset overlap' would be more informative with the numerical magnitude of the bias and a link to the generated trace.
  5. [Footnotes] Several URLs are broken by line breaks or spacing (e.g., the Semrush link in footnote 4). These should be fixed to ensure the reproducibility trail is accessible.

Circularity Check

0 steps flagged

No significant circularity: headline metrics are empirical benchmark results, not consequences of the framework's own definitions or self-citations.

full rationale

OGD4All's reported numbers are empirical outcomes of running 11 LLMs on a 199-question benchmark with manually authored ground-truth scripts; the paper contains no fitted parameter that is later renamed a prediction, no equation that reduces to its own input, and no uniqueness theorem imported from the authors' prior work. The accuracy/recall/answerability metrics are defined in Table I and measured directly. The potentially weakest point is the correctness judge: Section III-C says 'The correctness metric is computed using an LLM-as-a-judge approach... Preliminary experiments demonstrated that our LLM judge based on GPT-4o is more robust...' This is a validation gap and a correctness risk (especially since GPT-4o is itself one of the evaluated models), but it is not circular: the judge does not define the ground truth; the 169 reference answers and scripts were created independently, and no quantity is forced by construction. The paper also separates retrieval and analysis ('ground-truth relevant datasets are supplied for the analysis benchmark runs'), so the 98.2% analysis figure is conditional, not end-to-end—again a scoping limitation, not a circularity. Declared limitations in IV-D ('the quality of maps and plots was not quantitatively evaluated', 'behavior for ambiguous or open questions ... was not evaluated', benchmark saturation) are stated rather than concealed. The only overlapping-author citation ([1], Helbing et al.) supports a general e-governance claim in the introduction and is not load-bearing for the framework's results. No evidence of self-definitional reasoning, fitted-input-as-prediction, or renaming of a known result.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The central accuracy claims rest on a self-constructed benchmark, an LLM judge, and several hand-chosen engineering defaults (3 subqueries, 20 code steps, OpenAI embeddings, GPT-4o judge). None of these numbers were fitted against external data, but they are not derived from first principles either; changing them would change the reported scores.

free parameters (5)
  • max_subqueries = 3 (default)
    Chosen by hand in §III-A; caps retrieval reformulations and affects recall and cost.
  • max_analysis_steps = 20 (default)
    Chosen by hand in §III-B; caps code-generation iterations and affects error recovery and latency.
  • embedding_model = text-embedding-3-large
    Selected in §III-A; retrieval relevance depends on this commercial embedding model.
  • LLM_judge_model = GPT-4o
    Chosen in §III-C for robustness in scoring correctness; the evaluation metric depends on this model.
  • retrieval_top_k = not reported
    Number of nearest-neighbor metadata documents returned per subquery is not specified; affects retrieval recall.
axioms (5)
  • domain assumption The 169 ground-truth Python scripts and answers correctly encode the benchmark questions.
    Correctness is measured against these scripts (§III-C); incorrect ground truth invalidates all accuracy scores.
  • domain assumption GPT-4o as LLM-as-judge reliably equates model answers to reference answers across answer formats.
    Authors state preliminary experiments but provide no quantitative agreement or error analysis (§III-C).
  • domain assumption Semantic retrieval with text-embedding-3-large plus German query reformulation identifies all relevant datasets.
    Retrieval metrics depend on this design; no comparison to alternative embeddings or keyword baselines (§III-A).
  • domain assumption Code generation is superior to direct tool calling for this task.
    Design choice in §III-B based on cited prior work [13], [14], not re-validated in this paper.
  • domain assumption Unique-answer benchmark questions represent typical citizen OGD queries.
    Authors acknowledge ambiguity is common in geospatial QA and was not evaluated (§IV-D); this limits generalization.

reviewed 2026-08-03 · how reviews work

0 comments
Cite this review

Pith. "Pith review of OGD4All: A Framework for Accessible Interaction with Geospatial Open Government Data Based on Large Language Models." pith.science (2026). https://pith.science/paper/NKSYYCT6

@misc{pith2026260200012,
  author       = {Pith},
  title        = {Pith review of: OGD4All: A Framework for Accessible Interaction with Geospatial Open Government Data Based on Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NKSYYCT6}},
  note         = {Machine review of arXiv:2602.00012}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

We present OGD4All, a transparent, auditable, and reproducible framework based on Large Language Models (LLMs) to enhance citizens' interaction with geospatial Open Government Data (OGD). The system combines semantic data retrieval, agentic reasoning for iterative code generation, and secure sandboxed execution that produces verifiable multimodal outputs. Evaluated on a 199-question benchmark covering both factual and unanswerable questions, across 430 City-of-Zurich datasets and 11 LLMs, OGD4All reaches 98% analytical correctness and 94% recall while reliably rejecting questions unsupported by available data, which minimizes hallucination risks. Statistical robustness tests, as well as expert feedback, show reliability and social relevance. The proposed approach shows how LLMs can provide explainable, multimodal access to public data, advancing trustworthy AI for open governance.

Figures

Figures reproduced from arXiv: 2602.00012 by Dirk Helbing, Javier Argota S\'anchez-Vaquerizo, Krystian Samp, Luis Gisler, Michael Siebenmann, Stefan Arisona.

Figure 1
Figure 1. Figure 1: High-level overview of OGD4All, where an initial dataset retrieval stage is followed by a dataset analysis stage. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Screenshot of the Web-based prototype showcasing [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Benchmark dataset categories & frequent operations [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 3
Figure 3. Figure 3: Screenshot of the Web-based prototype. The retrieval [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Retrieval metrics across different LLMs. [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: shows the analysis performance of various LLMs. GPT-4.1 outperforms all other tested models in terms of correctness (98.2%). A clear performance gap is noticeable between the frontier closed-weight and open-weight models, with both Mistral models, Llama 4 Maverick, and GPT-oss 120B achieving only 60.9%-69.2% correctness, primarily due to lower reasoning and instruction-following abilities, and due to limit… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

26 extracted references · 2 canonical work pages

  1. [1]

    Democracy by Design: Perspectives for Digitally Assisted, Participatory Upgrades of Society,

    D. Helbing, S. Mahajan, R. H. Fricker, A. Musso, C. I. Hausladen, C. Carissimo, D. Carpentras, E. Stockinger, J. Argota Sanchez- Vaquerizo, J. C. Yang, M. C. Ballandies, M. Korecki, R. K. Dubey, and E. Pournaras, “Democracy by Design: Perspectives for Digitally Assisted, Participatory Upgrades of Society,”Journal of Computational Science, vol. 71, p. 1020...

    arXiv 2023

  2. [2]

    Toolformer: Language Models Can Teach Themselves to Use Tools,

    T. Schick, J. Dwivedi-Yu, R. Dessi, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom, “Toolformer: Language Models Can Teach Themselves to Use Tools,” inAdvances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36. Curran Associates, Inc., 2023, pp. 68 5...

    2023

  3. [3]

    Retrieval-augmented generation for knowledge-intensive NLP tasks,

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K ¨uttler, M. Lewis, W. T. Yih, T. Rockt ¨aschel, S. Riedel, and D. Kiela, “Retrieval-augmented generation for knowledge-intensive NLP tasks,” inAdvances in Neural Information Processing Systems, 2020, pp. 9459–9474. [Online]. Available: https://proceedings.neurip s.cc/paper/2020/hash/...

    2020

  4. [4]

    ReAct: Synergizing Reasoning and Acting in Language Models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao, “ReAct: Synergizing Reasoning and Acting in Language Models,” in11th International Conference on Learning Representations, ICLR 2023, mar 2023. [Online]. Available: https: //doi.org/10.48550/arXiv.2210.03629

    Pith/arXiv arXiv doi:10.48550/arxiv.2210.03629 2023

  5. [5]

    PAL: Program-aided language models,

    L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y . Yang, J. Callan, and G. Neubig, “PAL: Program-aided language models,” inProceedings of the 40th International Conference on Machine Learning, vol

  6. [6]

    Natural Language to Code Generation in Interactive Data Science Notebooks,

    P. Yin, W.-D. Li, K. Xiao, A. Rao, Y . Wen, K. Shi, J. Howland, P. Bailey, M. Catasta, H. Michalewski, O. Polozov, and C. Sutton, “Natural Language to Code Generation in Interactive Data Science Notebooks,” inProceedings of the 61st Annual Meeting of the Association for Computational Linguistics, vol. 1. Stroudsburg, PA, USA: Association for Computational...

    2023

  7. [7]

    A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions,

    L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, and T. Liu, “A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions,”ACM Transactions on Information Systems, vol. 43, no. 2, jan 2025. [Online]. Available: https://dl.acm.org/doi/10.1145/3703155

    doi:10.1145/3703155 2025

  8. [8]

    The uncertain relationship between open data and accountability: A response to Yu and Robinson’s The New Ambiguity of “Open Government

    T. Peixoto, “The uncertain relationship between open data and accountability: A response to Yu and Robinson’s The New Ambiguity of “Open Government”,”UCLA Law Review Discourse, vol. 60, no. 2007, pp. 200–213, 2013. [Online]. Available: https://heinonline.org/HOL/P?h=hein.journals/ucladis60&i=199

    2007

  9. [9]

    Designing a Large Language Model Based Open Data Assistant for Effective Use,

    T. C. Schelhorn, U. Gnewuch, and A. Maedche, “Designing a Large Language Model Based Open Data Assistant for Effective Use,” inLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 14621 LNCS. Springer Science and Business Media Deutschland GmbH, 2024, pp. 398–411. [Onlin...

    work page doi:10.1007/978-3-031-61175-9 2024

  10. [10]

    GeoGPT: An assistant for understanding and processing geospatial tasks,

    Y . Zhang, C. Wei, Z. He, and W. Yu, “GeoGPT: An assistant for understanding and processing geospatial tasks,”International Journal of Applied Earth Observation and Geoinformation, vol. 131, p. 103976, jul 2024. [Online]. Available: https://doi.org/10.1016/j.jag.20 24.103976

    work page doi:10.1016/j.jag.20 2024

  11. [11]

    UrbanLLM: Autonomous Urban Activity Planning and Management with Large Language Models,

    Y . Jiang, Q. Chao, Y . Chen, X. Li, S. Liu, and G. Cong, “UrbanLLM: Autonomous Urban Activity Planning and Management with Large Language Models,”arXiv preprint arXiv:2406.12360, jun 2024. [Online]. Available: https://arxiv.org/abs/2406.12360

    Pith/arXiv arXiv 2024

  12. [12]

    GeoTool-GPT: a trainable method for facilitating Large Language Models to master GIS tools,

    C. Wei, Y . Zhang, X. Zhao, Z. Zeng, Z. Wang, J. Lin, Q. Guan, and W. Yu, “GeoTool-GPT: a trainable method for facilitating Large Language Models to master GIS tools,”International Journal of Geographical Information Science, vol. 39, no. 4, pp. 707–731, 2025. [Online]. Available: https://doi.org/10.1080/13658816.2024.2438937

    arXiv 2025

  13. [13]

    If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agents,

    K. Yang, J. Liu, J. Wu, C. Yang, Y . R. Fung, S. Li, Z. Huang, X. Cao, X. Wang, Y . Wang, H. Ji, and C. Zhai, “If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agents,” arXiv preprint arXiv:2401.00812, jan 2024. [Online]. Available: https://arxiv.org/abs/2401.00812v2

    Pith/arXiv arXiv 2024

  14. [14]

    Executable code actions elicit better llm agents,

    X. Wang, Y . Chen, L. Yuan, Y . Zhang, Y . Li, H. Peng, and H. Ji, “Executable code actions elicit better llm agents,” in Proceedings of the 41st International Conference on Machine Learning, vol. 235. PMLR, 2024. [Online]. Available: https: //proceedings.mlr.press/v235/wang24h.html

    2024

  15. [15]

    Autonomous GIS: the next-generation AI- powered GIS,

    Z. Li and H. Ning, “Autonomous GIS: the next-generation AI- powered GIS,”International Journal of Digital Earth, vol. 16, no. 2, pp. 4668–4686, dec 2023. [Online]. Available: https: //doi.org/10.1080/17538947.2023.2278895

    arXiv 2023

  16. [16]

    ChatGeoAI: Enabling Geospatial Analysis for Public through Natural Language, with Large Language Models,

    A. Mansourian and R. Oucheikh, “ChatGeoAI: Enabling Geospatial Analysis for Public through Natural Language, with Large Language Models,”ISPRS International Journal of Geo-Information, vol. 13, no. 10, p. 348, oct 2024. [Online]. Available: https://doi.org/10.3390/ ijgi13100348

    2024

  17. [17]

    A Unified Framework of Five Principles for AI in Society,

    L. Floridi and J. Cowls, “A Unified Framework of Five Principles for AI in Society,”Harvard Data Science Review, vol. 1, no. 1, 6

  18. [18]

    Bridging the Gap Between Ethics and Practice,

    B. Shneiderman, “Bridging the Gap Between Ethics and Practice,” ACM Transactions on Interactive Intelligent Systems (TiiS), vol. 10, no. 4, p. 26, 10 2020. [Online]. Available: https://dl.acm.org/doi/10. 1145/3419764

    2020

  19. [19]

    The vocabulary problem in human-system communication,

    G. W. Furnas, T. K. Landauer, L. M. Gomez, and S. T. Dumais, “The vocabulary problem in human-system communication,” Communications of the ACM, vol. 30, no. 11, pp. 964–971, nov 1987. [Online]. Available: https://dl.acm.org/doi/10.1145/32206.32212

    arXiv 1987

  20. [20]

    MultiPL- E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation,

    F. Cassano, J. Gouwar, D. Nguyen, S. Nguyen, L. Phipps- Costin, D. Pinckney, M. H. Yee, Y . Zi, C. J. Anderson, M. Q. Feldman, A. Guha, M. Greenberg, and A. Jangda, “MultiPL- E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation,”IEEE Transactions on Software Engineering, vol. 49, no. 7, pp. 3675–3691, jul 2023. [Online]. Available: h...

    arXiv 2023

  21. [22]

    A survey on LLM-as-a- Judge,

    J. Gu, X. Jiang, Z. Shi, H. Tan, X. Zhai, C. Xu, W. Li, Y . Shen, S. Ma, H. Liu, S. Wang, K. Zhang, Z. Lin, B. Zhang, L. Ni, W. Gao, Y . Wang, and J. Guo, “A survey on LLM-as-a- Judge,”The Innovation, p. 101253, jan 2026. [Online]. Available: https://doi.org/10.1016/j.xinn.2025.101253

    arXiv 2026

  22. [23]

    Geographic Question Answering: Challenges, Uniqueness, Classification, and Future Directions,

    G. Mai, K. Janowicz, R. Zhu, L. Cai, and N. Lao, “Geographic Question Answering: Challenges, Uniqueness, Classification, and Future Directions,”AGILE: GIScience Series, vol. 2, pp. 1–21, jun

  23. [202]

    10 764–10 799, ISSN: 2640-3498

    PMLR, 2023, pp. 10 764–10 799, ISSN: 2640-3498. [Online]. Available: https://proceedings.mlr.press/v202/gao23f.html

    2023

  24. [2019]

    Available: https://hdsr.mitpress.mit.edu/pub/l0jsh9d1 /release/8

    [Online]. Available: https://hdsr.mitpress.mit.edu/pub/l0jsh9d1 /release/8

  25. [2021]

    Available: https://doi.org/10.5194/agile-giss-2-8-2021

    [Online]. Available: https://doi.org/10.5194/agile-giss-2-8-2021

    doi:10.5194/agile-giss-2-8-2021 2021

  26. [2025]

    Available: http://arxiv.org/abs/2507.07484

    [Online]. Available: http://arxiv.org/abs/2507.07484

    Pith/arXiv arXiv

This paper was first reviewed by deepseek-v4-flash on August 3, 2026.