{"id":"9961dcae-b408-4a9d-8768-70e7d6983f43","arxiv_id":"2412.12961","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"On a roughly 60-question Land Matrix benchmark, Codestral-22B with an agentic LLM pipeline generates the most syntactically valid REST and GraphQL queries, though overall accuracy remains modest.","lead":"The paper compares three open-weight language models and three prompting strategies for translating natural language questions into REST and GraphQL queries for the Land Matrix land-acquisition database. It finds Codestral-22B with an agentic pipeline works best, reaching about two-thirds of expected results on REST, but the evaluation is small-scale and statistically fragile.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1's 'Valid Query' column is defined as a Jaccard similarity in §4.1.1 but reported as syntactic validity; until this metric and its threshold are pinned down, the headline percentages are uninterpretable.","rationale":"The reader correctly identified the undefined threshold mapping and single-run evaluation as key weaknesses. My concern is more specific: the paper's own metric definitions in §4.1.1 do not match the labels on Table 1, so the headline 'Valid Query' numbers may not mean what the conclusion claims. This is an internal consistency issue rather than a disagreement with consensus, and it is fixable by documenting the evaluation code and threshold. Because the reader already marked the paper CONDITIONAL, my read does not change the verdict, but it adds a concrete condition that should be verified before the numbers are used. I do not see grounds for rejection: the experimental setup, corpus, and repository are described in enough detail that the metric can be checked, and the central comparison is plausible. The Discussion's claim of monotone improvement from prompt engineering to RAG to agents is contradicted by Table 1 for Codestral GraphQL (69% vs 56% versus 77%), but that is a secondary interpretive error, not the load-bearing flaw.","tokens_in":7312,"tokens_out":6282,"duration_ms":62848,"concrete_test":"Clone the released repository and inspect the evaluation script that produced Tables 1 and 4. Recompute Table 1 from the raw generated queries by submitting each query to the Land Matrix API for true syntactic validity, and separately recompute both tables using the Jaccard-similarity thresholds swept from 0.5 to 1.0. If the Valid Query column differs from the API-execution result, or if Codestral-Agentic is not consistently top-ranked across the threshold sweep, the central claim as stated is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing link between raw model outputs and the central conclusion is the binary classification of each generated query as 'Valid Query' and 'Valid Result' in Tables 1 and 4. Section 4.1.1 defines Valid Query by Jaccard similarity between the generated query and the expert query, and Valid Result by Jaccard similarity between the retrieved result sets, yet no similarity threshold is reported anywhere in the paper. Table 1 is then captioned 'Validity of the syntax of the requests', which is a different property: a query can be syntactically valid while being semantically dissimilar to the ground truth. If the table was produced by actual API execution, the metric definition in §4.1.1 is incorrect; if it was produced by a Jaccard threshold, then calling the resulting percentage 'syntactic validity' overstates what was measured. Every model comparison in the paper inherits these binary percentages, so an arbitrary threshold or an execution-vs-similarity mismatch can change the ranking. The GitHub repository is a positive step, but the manuscript itself must specify the mapping from Jaccard values to the reported percentages before the claim that Codestral-Agentic achieves 66% REST and 51% GraphQL valid results can be assessed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares three open-weight LLMs (Llama3-8B, Mixtral-8x7B-instruct, Codestral-22B) under three adaptations (prompt engineering, RAG, and an agentic pipeline) for translating natural-language questions into REST and GraphQL queries against the Land Matrix database. The authors built a corpus of about 60 real non-technical user questions with expert-written ground-truth queries, and they evaluate generated queries for syntactic validity, filter accuracy, and result-set coverage. The central empirical claim is that Codestral-22B in the agentic configuration performs best, achieving 52% valid REST queries and 77% valid GraphQL queries, with 66% and 51% valid results respectively. The paper also describes a reproducible GitHub repository and a Streamlit demonstration.","tokens_in":7524,"tokens_out":3189,"duration_ms":27554,"significance":"If the headline result holds, the paper provides a useful and honest applied benchmark for natural-language access to a specific real-world database, with an emphasis on reproducibility through a public repository and a working demonstrator. The use of real user questions and expert-configured ground-truth queries is a strength, and scoring against independent expert queries avoids direct circularity. The main value is practical: it identifies a promising configuration (code-specialized LLM with an entity-extraction agent) for the Land Matrix and similar structured-data portals. The contribution is empirical rather than methodological, and the evaluation design currently limits how strongly the comparison can be trusted.","major_comments":[{"comment":"The metric definition and the table caption are inconsistent. Valid Query is defined as Jaccard similarity between the generated query and the expert query, but Table 1 is captioned 'Validity of the syntax of the requests' and Section 5.1 interprets the numbers as syntactic validity. No threshold is given for converting Jaccard values into the reported percentages, and no indication is given of whether the percentages were obtained by executing the queries against the API or by thresholding similarity. Because every model comparison in Tables 1 and 4 inherits these binary percentages, the headline claim that Codestral-Agentic reaches 52% REST and 77% GraphQL 'valid queries' cannot be assessed until the authors specify the mapping from Jaccard values to the reported percentages and distinguish syntactic validity from semantic similarity to the ground truth.","section":"Section 4.1.1 and Table 1"},{"comment":"Each condition is evaluated with a single run of a stochastic system, and decoding parameters such as temperature, top-p, and random seed are not reported. With roughly 60 test questions, the difference between Codestral-Agentic (52% REST valid query) and Codestral-RAG (29%) may correspond to a handful of questions, so the ranking is not statistically grounded. The authors should report per-condition raw counts, multiple runs, or confidence intervals, or at minimum disclose the decoding settings and seed so that the single-run comparison is reproducible.","section":"Section 5 (Tables 1-4)"},{"comment":"Filter accuracy is computed only on syntactically valid queries, but the tables do not report the number of valid queries per cell. For example, Codestral prompt-engineering REST has 2% valid queries in Table 1, so a 100% precision / 100% recall row likely rests on one observation; similarly, Llama3 agentic REST (2% valid) yields 100% precision but only 36% recall. The paper should give per-cell denominators or present raw counts for sparse cells, since percentages based on one or two queries are not comparable to those based on dozens.","section":"Section 5.2 (Tables 2 and 3)"},{"comment":"The RAG configuration does not report the top-k value or the Faiss retrieval settings, and the agentic configuration does not specify the model and prompt used for the entity-extraction agent. Because the paper advertises reproducibility, these parameters should be stated in the manuscript itself (or the repository version should be pinned with a clear commit), rather than left to the reader to infer from the code.","section":"Sections 3.2 and 3.3"}],"minor_comments":[{"comment":"The sentence 'These include metrics for evaluating the overall quality of the queries and those for assessing the attributes used within them like.' is incomplete and should be rewritten.","section":"Section 4.1"},{"comment":"The table header contains 'recallL' and 'accuracy' with inconsistent formatting, and the dash entries are not defined in a caption or footnote.","section":"Tables 2 and 3"},{"comment":"The discussion paragraph contains the typo 'GrapQL' instead of 'GraphQL'.","section":"Section 5.1"},{"comment":"The reference 'Pedro et al. (2023)' appears as a standalone line immediately before the description of the demonstrator; it is not integrated into the surrounding text.","section":"Section 7"},{"comment":"The paper says the corpus contains 'almost 60' questions but does not report the exact corpus size, the number of REST versus GraphQL questions, or the train/test split used for the evaluation.","section":"Section 3"},{"comment":"Figure 1 is referenced in Section 3.1 but the figure content is not included in the manuscript text; the prompt structure should be visible or fully described in prose.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a solid applied comparison for a specific database, and the authors are transparent about the goal of reproducibility. The main concern is that the metric ambiguity and the single-run evaluation make the headline numbers difficult to verify, so I would ask for a revised version that specifies thresholds, decoding parameters, and per-cell counts. The contribution is more empirical than methodological, so the editor should weigh fit with the journal's scope accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful core here is a new benchmark: just under 60 real Land Matrix user questions with expert-written REST and GraphQL queries, plus a systematic comparison of three open-weight LLMs under prompt, RAG, and agentic adaptations. The GitHub repo and demo are real assets, and the finding that a code-specialized model (Codestral-22B) in an agentic setup works best is plausible and consistent with the filter-accuracy tables. That is a legitimate applied contribution to a domain that could use more entry points.\n\nThe soft spot is the metric. Section 4.1.1 defines Valid Query as Jaccard similarity between generated and expert queries, but Table 1 reports it as \"validity of the syntax.\" Those are different properties: a query can be syntactically correct yet token-wise dissimilar from the ground truth. No threshold is given for turning a Jaccard score into a binary valid/invalid label. If the queries were actually executed against the API, then the correct metric is execution success, not similarity. Right now the headline percentages are uninterpretable, and every model comparison inherits that ambiguity.\n\nBeyond that, the evaluation is statistically thin: one run per condition, about sixty questions, no error bars, no decoding parameters, and RAG top-k is never stated. The Discussion also overstates the case, claiming performance improves from prompt to RAG to agents, but the tables do not consistently show that. For example, Codestral GraphQL validity drops from 69% with prompt engineering to 56% with RAG, and Llama3 drops from 65% to 49%.\n\nThe good news is that the core comparison is likely salvageable. The agentic pipeline does look stronger on filter precision and valid-result accuracy, and the benchmark itself is worth having. But the manuscript needs a major revision before the numbers can be trusted for tool selection: define the metrics precisely, report the threshold and decoding settings, run multiple seeds, and correct the RAG discussion.\n\nI would send this to peer review rather than desk-rejecting it, but with the expectation of heavy revision. The benchmark contribution and reproducibility are enough to warrant referee time.","headline":"A useful Land Matrix benchmark and reproducible comparison, but the headline validity metric is under-specified and the paper overreads its own tables.","tokens_in":8097,"tokens_out":1884,"would_cite":true,"duration_ms":19060,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"On Land Matrix data, a code-focused model with an entity-extraction agent turns plain-language questions into correct queries for two-thirds of REST requests and half of GraphQL requests.","keywords":["natural language to query","REST API","GraphQL","large language models","retrieval-augmented generation","LLM agents","Land Matrix","land acquisitions"],"falsifier":"Run the Codestral-Agentic configuration dozens of times on the same 60 questions with non-zero temperature and plot the distribution of Valid Result for REST; if the 66% figure fluctuates by more than the gap to the second-best configuration, the paper's headline comparison is not stable.","tokens_in":7088,"feed_emoji":"🤖","tokens_out":9517,"duration_ms":78752,"temperature":0.7,"pith_summary":"This paper tests whether large language models, adapted with prompt engineering, retrieval-augmented generation (RAG), or a two-agent pipeline, can let non-technical users query the Land Matrix database in natural language without learning REST or GraphQL syntax. The authors built a corpus of almost 60 real user questions with expert-written reference queries and compared three open-weight models. They report that the code-specialized model in the agentic configuration is clearly the best: it produces syntactically valid requests 52% of the time for REST and 77% for GraphQL, and returns the expected data 66% of the time for REST and 51% for GraphQL. The practical message is that an entity-extraction front end plus a code-oriented model is the most promising route to natural-language access to this database.","feed_headline":"Code model plus agent answers 2 in 3 natural-language data queries","feed_subtitle":"On 60 real user questions, Codestral with an entity extractor beat larger models on both REST and GraphQL accuracy.","key_machinery":"The central mechanism is the two-agent pipeline: a first agent extracts [entity: value] pairs from the user's question using predefined values in the Land Matrix database, and a second agent generates the REST or GraphQL query using those values, a small set of similar question-query pairs retrieved by RAG, the database schema, and API rules. This targeted context avoids dumping the full schema into the prompt, which the paper argues overwhelms the model. Evaluation relies on Jaccard similarity $J(A,B)=|A\\cap B|/|A\\cup B|$ between generated and reference queries and between their API results, with valid-query and valid-result thresholds on top of that similarity.","core_discovery":"The paper claims that the winning combination is a code-specialized large language model paired with an entity-extraction agent that pre-fills known attribute values before the query-writing step. On the roughly 60-question corpus, this setup reaches 52% syntactic validity for REST and 77% for GraphQL, and 66% valid results for REST versus 51% for GraphQL, which the authors describe as two-thirds of the results expected by database users for REST and half for GraphQL. The same evaluation shows agentic adaptation beating prompt engineering, and prompt engineering beating RAG, with RAG sometimes hurting performance by adding irrelevant retrieved context. The authors frame the contribution as an extensible, reproducible comparison rather than a deployed product.","pith_inferences":["The reported success rates imply that a practical natural-language interface to Land Matrix would still need a verification or repair step before showing results, since about one REST question in three fails even with the best tested setup.","The entity-extraction-then-query-generation split should transfer to other structured databases whose main difficulty is mapping user words to canonical attribute values rather than query syntax.","The single-run, roughly 60-question protocol means the exact percentages are point estimates; a stratified benchmark with more questions would reveal which question types drive the failures.","RAG underperforming both simpler prompt engineering and the agentic setup suggests that retrieved similar examples can distract the generator when the schema is already present, a hypothesis the paper does not test directly."],"forward_implications":["Agentic adaptation outperforms prompt engineering and RAG across models and metrics, and is the only configuration that makes REST query generation usable at all.","Codestral-22B, specialized for code, dominates the general instruction-tuned models on both syntax validity and filter accuracy, so code specialization matters more than model size in this task.","Even the best setup returns correct data for only 66% of REST questions, so roughly one-third of natural-language REST requests still fail end-to-end today.","GraphQL syntax is consistently easier for all models than REST syntax, indicating that the request format is a bigger barrier than user phrasing.","Because the code and data are released, new models can be added to the same benchmark and compared directly on the same corpus."],"supporting_citations":[{"why":"It defines the Land Matrix database and the 200-hectare threshold that shapes the reference queries.","marker":"Anseeuw et al., 2012"},{"why":"It supplies the Text-to-SQL framing that the paper adapts to REST and GraphQL generation.","marker":"Hong et al., 2024"},{"why":"It provides the few-shot prompting technique used in the prompt-engineering adaptation.","marker":"Brown et al., 2020"},{"why":"It provides the retrieval-augmented generation method used to pull similar question-query pairs into the prompt.","marker":"Lewis et al., 2021"},{"why":"It motivates the choice of a code-specialized model by showing that code-focused training improves code generation.","marker":"Rozière et al., 2024"},{"why":"It supplies the advanced text-to-SQL prompting ideas that the paper compares against.","marker":"Sun et al., 2024"}],"fun_headline_variants":["Code LLM plus entity agent tops LandMatrix query tests","Entity extraction agent boosts code model's database query accuracy","Small code-specialized AI outperforms larger models on land data","Agent combo lifts natural-language query success on land data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline comparison rests on about 60 expert-written question-query pairs being representative of real user intent, and on single-run scores from each stochastic model at undisclosed decoding settings, so unrepresentative questions or noisy runs could reshuffle the ranking.","fun_headline_variants_meta":{"raw":{"variants":["Code LLM plus entity agent tops LandMatrix query tests","Entity extraction agent boosts code model's database query accuracy","Small code-specialized AI outperforms larger models on land data","Agent combo lifts natural-language query success on land data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000679,"raw_usage":{"total_tokens":3046,"prompt_tokens":863,"completion_tokens":2183,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":479,"completion_tokens_details":{"reasoning_tokens":2116}},"tokens_in":479,"tokens_out":2183,"duration_ms":16460,"temperature":1.0,"reasoning_tokens":2116,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:32:27.891651+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the Codestral-Agentic configuration dozens of times on the same 60 questions with non-zero temperature and plot the distribution of Valid Result for REST; if the 66% figure fluctuates by more than the gap to the second-best configuration, the paper's headline comparison is not stable.","supporting_citations":[{"cited_title":"Boche, T","cited_arxiv_id":null,"evidence_quote":"It defines the Land Matrix database and the 200-hectare threshold that shapes the reference queries."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies the Text-to-SQL framing that the paper adapts to REST and GraphQL generation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies the advanced text-to-SQL prompting ideas that the paper compares against."}],"review_version":1}