Adaptations of AI models for querying the LandMatrix database in natural language

Fatiha Ait Kbir; J\'er\'emy Bourgoin; Marie Gradeler; R\'emy Decoupes; Roberto Interdonato

REVIEW 4 major objections 6 minor 13 references

Adaptations of AI models for querying the LandMatrix database in natural language

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read On Land Matrix data, a code-focused model with an entity-extraction agent turns plain-language questions into correct queries for two-thirds of REST requests and half of GraphQL requests.

desk verdict A useful Land Matrix benchmark and reproducible comparison, but the headline validity metric is under-specified and the paper overreads its own tables. read the letter →

arxiv 2412.12961 v1 pith:PSGTBKSF submitted 2024-12-17 cs.CL

Fatiha Ait Kbir , Jérémy Bourgoin , Rémy Decoupes , Marie Gradeler , Roberto Interdonato This is my paper · ORCID

classification cs.CL

keywords naturallanguagetoqueryRESTAPIGraphQLlargemodelsretrieval-augmentedgenerationLLMagentsLandMatrixacquisitions

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tests whether large language models, adapted with prompt engineering, retrieval-augmented generation (RAG), or a two-agent pipeline, can let non-technical users query the Land Matrix database in natural language without learning REST or GraphQL syntax. The authors built a corpus of almost 60 real user questions with expert-written reference queries and compared three open-weight models. They report that the code-specialized model in the agentic configuration is clearly the best: it produces syntactically valid requests 52% of the time for REST and 77% for GraphQL, and returns the expected data 66% of the time for REST and 51% for GraphQL. The practical message is that an entity-extraction front end plus a code-oriented model is the most promising route to natural-language access to this database.

What carries the argument

The central mechanism is the two-agent pipeline: a first agent extracts [entity: value] pairs from the user's question using predefined values in the Land Matrix database, and a second agent generates the REST or GraphQL query using those values, a small set of similar question-query pairs retrieved by RAG, the database schema, and API rules. This targeted context avoids dumping the full schema into the prompt, which the paper argues overwhelms the model. Evaluation relies on Jaccard similarity $J(A,B)=|A\cap B|/|A\cup B|$ between generated and reference queries and between their API results, with valid-query and valid-result thresholds on top of that similarity.

What would settle it

Run the Codestral-Agentic configuration dozens of times on the same 60 questions with non-zero temperature and plot the distribution of Valid Result for REST; if the 66% figure fluctuates by more than the gap to the second-best configuration, the paper's headline comparison is not stable.

Watch

Extended reading notes

Core claim

The paper claims that the winning combination is a code-specialized large language model paired with an entity-extraction agent that pre-fills known attribute values before the query-writing step. On the roughly 60-question corpus, this setup reaches 52% syntactic validity for REST and 77% for GraphQL, and 66% valid results for REST versus 51% for GraphQL, which the authors describe as two-thirds of the results expected by database users for REST and half for GraphQL. The same evaluation shows agentic adaptation beating prompt engineering, and prompt engineering beating RAG, with RAG sometimes hurting performance by adding irrelevant retrieved context. The authors frame the contribution as an extensible, reproducible comparison rather than a deployed product.

Load-bearing premise

The headline comparison rests on about 60 expert-written question-query pairs being representative of real user intent, and on single-run scores from each stochastic model at undisclosed decoding settings, so unrepresentative questions or noisy runs could reshuffle the ranking.

Editorial extensions

If this is right

Agentic adaptation outperforms prompt engineering and RAG across models and metrics, and is the only configuration that makes REST query generation usable at all.
Codestral-22B, specialized for code, dominates the general instruction-tuned models on both syntax validity and filter accuracy, so code specialization matters more than model size in this task.
Even the best setup returns correct data for only 66% of REST questions, so roughly one-third of natural-language REST requests still fail end-to-end today.
GraphQL syntax is consistently easier for all models than REST syntax, indicating that the request format is a bigger barrier than user phrasing.
Because the code and data are released, new models can be added to the same benchmark and compared directly on the same corpus.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The reported success rates imply that a practical natural-language interface to Land Matrix would still need a verification or repair step before showing results, since about one REST question in three fails even with the best tested setup.
The entity-extraction-then-query-generation split should transfer to other structured databases whose main difficulty is mapping user words to canonical attribute values rather than query syntax.
The single-run, roughly 60-question protocol means the exact percentages are point estimates; a stratified benchmark with more questions would reveal which question types drive the failures.
RAG underperforming both simpler prompt engineering and the agentic setup suggests that retrieved similar examples can distract the generator when the schema is already present, a hypothesis the paper does not test directly.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The useful core here is a new benchmark: just under 60 real Land Matrix user questions with expert-written REST and GraphQL queries, plus a systematic comparison of three open-weight LLMs under prompt, RAG, and agentic adaptations. The GitHub repo and demo are real assets, and the finding that a code-specialized model (Codestral-22B) in an agentic setup works best is plausible and consistent with the filter-accuracy tables. That is a legitimate applied contribution to a domain that could use more entry points.

The soft spot is the metric. Section 4.1.1 defines Valid Query as Jaccard similarity between generated and expert queries, but Table 1 reports it as "validity of the syntax." Those are different properties: a query can be syntactically correct yet token-wise dissimilar from the ground truth. No threshold is given for turning a Jaccard score into a binary valid/invalid label. If the queries were actually executed against the API, then the correct metric is execution success, not similarity. Right now the headline percentages are uninterpretable, and every model comparison inherits that ambiguity.

Beyond that, the evaluation is statistically thin: one run per condition, about sixty questions, no error bars, no decoding parameters, and RAG top-k is never stated. The Discussion also overstates the case, claiming performance improves from prompt to RAG to agents, but the tables do not consistently show that. For example, Codestral GraphQL validity drops from 69% with prompt engineering to 56% with RAG, and Llama3 drops from 65% to 49%.

The good news is that the core comparison is likely salvageable. The agentic pipeline does look stronger on filter precision and valid-result accuracy, and the benchmark itself is worth having. But the manuscript needs a major revision before the numbers can be trusted for tool selection: define the metrics precisely, report the threshold and decoding settings, run multiple seeds, and correct the RAG discussion.

I would send this to peer review rather than desk-rejecting it, but with the expectation of heavy revision. The benchmark contribution and reproducibility are enough to warrant referee time.

Referee Report

4 major / 6 minor

Summary. The paper compares three open-weight LLMs (Llama3-8B, Mixtral-8x7B-instruct, Codestral-22B) under three adaptations (prompt engineering, RAG, and an agentic pipeline) for translating natural-language questions into REST and GraphQL queries against the Land Matrix database. The authors built a corpus of about 60 real non-technical user questions with expert-written ground-truth queries, and they evaluate generated queries for syntactic validity, filter accuracy, and result-set coverage. The central empirical claim is that Codestral-22B in the agentic configuration performs best, achieving 52% valid REST queries and 77% valid GraphQL queries, with 66% and 51% valid results respectively. The paper also describes a reproducible GitHub repository and a Streamlit demonstration.

Significance. If the headline result holds, the paper provides a useful and honest applied benchmark for natural-language access to a specific real-world database, with an emphasis on reproducibility through a public repository and a working demonstrator. The use of real user questions and expert-configured ground-truth queries is a strength, and scoring against independent expert queries avoids direct circularity. The main value is practical: it identifies a promising configuration (code-specialized LLM with an entity-extraction agent) for the Land Matrix and similar structured-data portals. The contribution is empirical rather than methodological, and the evaluation design currently limits how strongly the comparison can be trusted.

major comments (4)

[Section 4.1.1 and Table 1] The metric definition and the table caption are inconsistent. Valid Query is defined as Jaccard similarity between the generated query and the expert query, but Table 1 is captioned 'Validity of the syntax of the requests' and Section 5.1 interprets the numbers as syntactic validity. No threshold is given for converting Jaccard values into the reported percentages, and no indication is given of whether the percentages were obtained by executing the queries against the API or by thresholding similarity. Because every model comparison in Tables 1 and 4 inherits these binary percentages, the headline claim that Codestral-Agentic reaches 52% REST and 77% GraphQL 'valid queries' cannot be assessed until the authors specify the mapping from Jaccard values to the reported percentages and distinguish syntactic validity from semantic similarity to the ground truth.
[Section 5 (Tables 1-4)] Each condition is evaluated with a single run of a stochastic system, and decoding parameters such as temperature, top-p, and random seed are not reported. With roughly 60 test questions, the difference between Codestral-Agentic (52% REST valid query) and Codestral-RAG (29%) may correspond to a handful of questions, so the ranking is not statistically grounded. The authors should report per-condition raw counts, multiple runs, or confidence intervals, or at minimum disclose the decoding settings and seed so that the single-run comparison is reproducible.
[Section 5.2 (Tables 2 and 3)] Filter accuracy is computed only on syntactically valid queries, but the tables do not report the number of valid queries per cell. For example, Codestral prompt-engineering REST has 2% valid queries in Table 1, so a 100% precision / 100% recall row likely rests on one observation; similarly, Llama3 agentic REST (2% valid) yields 100% precision but only 36% recall. The paper should give per-cell denominators or present raw counts for sparse cells, since percentages based on one or two queries are not comparable to those based on dozens.
[Sections 3.2 and 3.3] The RAG configuration does not report the top-k value or the Faiss retrieval settings, and the agentic configuration does not specify the model and prompt used for the entity-extraction agent. Because the paper advertises reproducibility, these parameters should be stated in the manuscript itself (or the repository version should be pinned with a clear commit), rather than left to the reader to infer from the code.

minor comments (6)

[Section 4.1] The sentence 'These include metrics for evaluating the overall quality of the queries and those for assessing the attributes used within them like.' is incomplete and should be rewritten.
[Tables 2 and 3] The table header contains 'recallL' and 'accuracy' with inconsistent formatting, and the dash entries are not defined in a caption or footnote.
[Section 5.1] The discussion paragraph contains the typo 'GrapQL' instead of 'GraphQL'.
[Section 7] The reference 'Pedro et al. (2023)' appears as a standalone line immediately before the description of the demonstrator; it is not integrated into the surrounding text.
[Section 3] The paper says the corpus contains 'almost 60' questions but does not report the exact corpus size, the number of REST versus GraphQL questions, or the train/test split used for the evaluation.
[Figure 1] Figure 1 is referenced in Section 3.1 but the figure content is not included in the manuscript text; the prompt structure should be visible or fully described in prose.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: model outputs are scored against independent expert-written ground-truth queries, and no central claim reduces to a fitted input or self-citation.

full rationale

The paper's derivation chain is an empirical benchmark: real user questions with expert-written REST/GraphQL queries are used as ground truth; LLMs generate queries under three adaptations; generated queries are executed and compared to ground truth by Jaccard similarity of result sets and filter accuracy. None of the reported percentages are defined in terms of the model's own outputs, and no parameter is fitted to the test set and then renamed as a prediction. The corpus is described as split into a part used to give the LLM query structure and a test part, which is the standard protection against RAG leakage; although the RAG section does not explicitly state that the retrieval index excludes test questions, the split statement makes the opposite interpretation unsupported. The mismatch between the Jaccard-based 'Valid Query' definition in Section 4.1.1 and the 'syntactic validity' caption in Table 1, and the absence of a stated similarity threshold, are threats to interpretability and correctness, not circularity. References are standard external work; no load-bearing self-citation or author-imported uniqueness claim appears. The central conclusion therefore has independent empirical content, so the circularity score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the quality of a small expert-annotated benchmark, on undisclosed evaluation thresholds and decoding settings, and on the completeness of the schema information. No new entities are introduced. The absence of a fixed public benchmark artifact and the lack of statistical repetition are the main ledger entries.

free parameters (3)

RAG top-k = not reported
The number of similar question-query pairs injected into the prompt is chosen by hand and is not disclosed, directly affecting RAG performance.
Validity threshold for Jaccard-based metrics = not reported
Tables 1 and 4 report percentages for 'Valid Query' and 'Valid Result' without defining the similarity cutoff that separates valid from invalid.
LLM decoding parameters = not reported
Temperature and other decoding settings for stochastic LLM generation are not given, yet they can change syntactic validity substantially.

assumptions (3)

domain assumption Expert-configured queries in the corpus are correct ground-truth interpretations of user intents.
All metrics compare generated queries to these expert queries; annotation errors would bias every reported score.
domain assumption The roughly 60-question corpus is representative of real non-technical user requests to Land Matrix.
Section 3 states partners gathered real requests, but no sampling protocol or coverage analysis is given, so generalizability is assumed.
domain assumption The Land Matrix API schema and value lists used in prompts are accurate and complete.
Section 3.1 says the prompt includes a list of all possible attributes and values; any omission or stale entry would penalize all models equally but still distort absolute accuracy.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Adaptations of AI models for querying the LandMatrix database in natural language." pith.science (2026). https://pith.science/paper/PSGTBKSF

@misc{pith2026241212961,
  author       = {Pith},
  title        = {Pith review of: Adaptations of AI models for querying the LandMatrix database in natural language},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PSGTBKSF}},
  note         = {Machine review of arXiv:2412.12961}
}

read the original abstract

The Land Matrix initiative (https://landmatrix.org) and its global observatory aim to provide reliable data on large-scale land acquisitions to inform debates and actions in sectors such as agriculture, extraction, or energy in low- and middle-income countries. Although these data are recognized in the academic world, they remain underutilized in public policy, mainly due to the complexity of access and exploitation, which requires technical expertise and a good understanding of the database schema. The objective of this work is to simplify access to data from different database systems. The methods proposed in this article are evaluated using data from the Land Matrix. This work presents various comparisons of Large Language Models (LLMs) as well as combinations of LLM adaptations (Prompt Engineering, RAG, Agents) to query different database systems (GraphQL and REST queries). The experiments are reproducible, and a demonstration is available online: https://github.com/tetis-nlp/landmatrix-graphql-python.

Figures

Figures reproduced from arXiv: 2412.12961 by the authors.

**Figure 2.** Pipeline of the Agents optimization: another language model is used to extract the filters wanted by the user [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 3 canonical work pages

[1]

Boche, T

Anseeuw, W., M. Boche, T. Breu, M. Giger, J. Lay, P. Messerli, and K. Nolte (2012). Transnational land deals for agriculture in the global south: analytical report based on the Land Matrix database . CDE. CIRAD-ES-UMR ART-DEV (ZAF); CIRAD-ES-UMR ART-DEV (FRA)

work page 2012
[2]

Benara, R

Balaguer, A., V . Benara, R. L. d. F. Cunha, R. d. M. E. Filho, T. Hendry, D. Holstein, J. Marsman, N. Mecklenburg, S. Malvar, L. O. Nunes, R. Padilha, M. Sharp, B. Silva, S. Sharma, V . Aski, and R. Chandra (2024). RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture. arXiv:2401.08406 [cs]

arXiv 2024
[3]

Brown, T. B., B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Am...

arXiv 2020
[4]

Chang, K

Devlin, J., M.-W. Chang, K. Lee, and K. Toutanova (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 [cs]. arXiv: 1810.04805

arXiv 2019
[5]

Hong, Z., Z. Yuan, Q. Zhang, H. Chen, J. Dong, F. Huang, and X. Huang (2024). Next-generation database interfaces: A survey of LLM-based text-to-SQL

work page 2024
[6]

Perez, A

Lewis, P., E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rock- täschel, S. Riedel, and D. Kiela (2021). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv:2005.11401 [cs]

arXiv 2021
[7]

Liu, S., J. Xu, W. Tjangnaka, S. J. Semnani, C. J. Yu, and M. S. Lam (2024). SUQL: Conversational Search over Structured and Unstructured Data with Large Language Models. arXiv:2311.09818 [cs]

arXiv 2024
[8]

Castro, P

Pedro, R., D. Castro, P. Carreira, and N. Santos (2023). From prompt injections to SQL injection attacks: How protected is your LLM-integrated web application? Rozière, B., J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, R. Sauvestre, T. Remez, J. Rapin, A. Kozhevnikov, I. Evtimov, J. Bitton, M. Bhatt, C. C. Ferrer, A. Grattafiori,...

arXiv 2023

Show all 13 references

[9]

Sun, R., S. O. Arik, A. Muzio, L. Miculicich, S. Gundabathula, P. Yin, H. Dai, H. Nakhost, R. Sinha, Z. Wang, and T. Pfister (2024). SQL-PaLM: Improved large language model adaptation for text-to-SQL (extended)

2024
[10]

Shazeer, N

Vaswani, A., N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin (2017). Attention Is All You Need. Number: arXiv:1706.03762 arXiv:1706.03762 [cs]

2017 arXiv
[11]

Yang, J., H. Jin, R. Tang, X. Han, Q. Feng, H. Jiang, B. Yin, and X. Hu (2023). Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond. arXiv:2304.13712 [cs]

2023 arXiv
[12]

Zhao, W. X., K. Zhou, J. Li, T. Tang, X. Wang, Y . Hou, Y . Min, B. Zhang, J. Zhang, Z. Dong, Y . Du, C. Yang, Y . Chen, Z. Chen, J. Jiang, R. Ren, Y . Li, X. Tang, Z. Liu, P. Liu, J.-Y . Nie, and J.-R. Wen (2023). A Survey of Large Language Models. arXiv:2303.18223 [cs]

2023 arXiv
[13]

Chiang, Y

Zheng, L., W.-L. Chiang, Y . Sheng, T. Li, S. Zhuang, Z. Wu, Y . Zhuang, Z. Li, Z. Lin, E. P. Xing, J. E. Gonza- lez, I. Stoica, and H. Zhang (2023). LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset. arXiv:2309.11998 [cs]. 8

2023 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

Boche, T

Anseeuw, W., M. Boche, T. Breu, M. Giger, J. Lay, P. Messerli, and K. Nolte (2012). Transnational land deals for agriculture in the global south: analytical report based on the Land Matrix database . CDE. CIRAD-ES-UMR ART-DEV (ZAF); CIRAD-ES-UMR ART-DEV (FRA)

work page 2012

[2] [2]

Benara, R

Balaguer, A., V . Benara, R. L. d. F. Cunha, R. d. M. E. Filho, T. Hendry, D. Holstein, J. Marsman, N. Mecklenburg, S. Malvar, L. O. Nunes, R. Padilha, M. Sharp, B. Silva, S. Sharma, V . Aski, and R. Chandra (2024). RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture. arXiv:2401.08406 [cs]

arXiv 2024

[3] [3]

Brown, T. B., B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Am...

arXiv 2020

[4] [4]

Chang, K

Devlin, J., M.-W. Chang, K. Lee, and K. Toutanova (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 [cs]. arXiv: 1810.04805

arXiv 2019

[5] [5]

Hong, Z., Z. Yuan, Q. Zhang, H. Chen, J. Dong, F. Huang, and X. Huang (2024). Next-generation database interfaces: A survey of LLM-based text-to-SQL

work page 2024

[6] [6]

Perez, A

Lewis, P., E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rock- täschel, S. Riedel, and D. Kiela (2021). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv:2005.11401 [cs]

arXiv 2021

[7] [7]

Liu, S., J. Xu, W. Tjangnaka, S. J. Semnani, C. J. Yu, and M. S. Lam (2024). SUQL: Conversational Search over Structured and Unstructured Data with Large Language Models. arXiv:2311.09818 [cs]

arXiv 2024

[8] [8]

Castro, P

Pedro, R., D. Castro, P. Carreira, and N. Santos (2023). From prompt injections to SQL injection attacks: How protected is your LLM-integrated web application? Rozière, B., J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, R. Sauvestre, T. Remez, J. Rapin, A. Kozhevnikov, I. Evtimov, J. Bitton, M. Bhatt, C. C. Ferrer, A. Grattafiori,...

arXiv 2023