REVIEW 3 major objections 5 minor 10 references
Chatting with your ERP: A Recipe
T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read An open-weight LLM agent turns natural-language ERP questions into working SQL, answering 10 of 11 expert-validated queries.
desk verdict A practical but overclaimed ERP Text2SQL recipe: honest engineering, unsupported 'novel dual-agent' claim, and an evaluation that never tests the central architecture. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is the SQL Agent's two-role loop: the Reasoner builds a first SQL statement from the user's intent and a Markdown schema, executes it, reads the error message, and reworks the query for up to N attempts; the Critic then checks syntactic validity, semantic appropriateness, readability, efficiency, and a subset of returned rows, and sends structured feedback back to the Reasoner for up to M refinement rounds. Around this loop sit a REACT-based conversational agent with a human-in-the-loop intent check, a hybrid reasoning-extraction pipeline that uses a small structured-output model to pull a formatted SQL block out of the reasoner's Markdown, and a two-part schema injection made of an expert's natural-language description plus automatically extracted tables, columns, types, and sample values.
What would settle it
Give the same eleven questions to two or more ERP experts independently and compare their verdicts against the SQL the agent produced; if the experts disagree on several answers, or if re-running the same questions with single-pass prompting matches the 10/11 score, the reasoner/critic loop is not doing the work the paper attributes to it.
Extended reading notes
Core claim
The central claim is that the reliability problem in LLM-based Text-to-SQL is best attacked at the agent level, not the model level: a reasoner that iterates against execution errors plus a critic that checks semantic fit, readability, efficiency, and a sample of output rows produces markedly more correct queries on a real enterprise schema than a single-pass prompt would. The strongest result is Devstral 24B Q4 reaching 10/11 and Qwen 2.5 32B Q4 reaching 9/11 on realistic domain queries, while several comparable open-weight coding models scored 0/11. The paper also shows that the schema has to be partially hand-documented: only 119 of 321 columns had natural-language descriptions, and most table relationships were not declared as foreign keys, so the handcrafted semantic description carries much of the load.
Load-bearing premise
The reported accuracy rests entirely on eleven (the text elsewhere says twelve) expert-written questions and a single expert's binary correct-or-incorrect judgment, with no rubric and no second judge; if that test set is unrepresentative or the judgment is inconsistent, the paper's central reliability claim collapses.
Editorial extensions
If this is right
- Organizations can build a natural-language ERP interface entirely on local, open-weight models, avoiding sending proprietary business data to external API providers.
- The recipe's five setup steps—read-only database exposure, documentation of business-critical fields, an expert-written semantic description, LLM selection, and a small model for structured extraction—are presented as sufficient to reproduce the agent.
- Model choice dominates outcomes even within a fixed architecture: Devstral 24B Q4 and Qwen 2.5 32B Q4 succeeded on most questions while Codestral 22B Q4, Magistral 24B Q4, and Deepseek Coder 33B Q4 all failed, so the loop alone does not guarantee quality.
- The two named bottlenecks for adoption are the manual expert-written schema, which blocks plug-and-play use, and the agent's difficulty choosing the right columns when a prompt is vague or IDs are ambiguous.
- Scale and latency constrain practical deployment: two of the attempted models could not finish inference within the three-minute limit on the available hardware.
Reading between the lines
- Editorial inference: the 10/11 and 9/11 scores are likely upper bounds on production reliability, because the test set is small, unreleased, and judged by a single expert; a larger question set with multiple independent judges would probably compress the spread between models.
- Editorial inference: the critic sees the same schema as the reasoner, so its semantic checks are only as good as the hand-written documentation; gains from the critique stage should be largest on syntax and execution errors and smallest on cryptic ID columns whose meaning only a human domain expert knows.
- Editorial inference: a fair comparison with single-pass prompting on the same eleven questions is the natural next experiment; if single-pass Devstral also scores near 10/11, the contribution of the reasoner/critic loop would be small.
- Editorial inference: because the prompts were built around Qwen 2.5 32B, the 0/11 results for other models may reflect prompt mismatch rather than model inability, and prompt tuning per model could narrow the gaps.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper describes an LLM-based agent for natural-language access to a production ERP database. It uses a REACT orchestrator, a two-agent SQL pipeline (reasoner plus critic), a handcrafted semantic schema, structured output extraction, and human-in-the-loop clarification. Nine open-weight models are evaluated on a small expert-validated query set that is reported as twelve queries in §4.2 and eleven in §5, with Devstral 24B Q4 achieving 10/11 and Qwen 2.5 32B Q4 achieving 9/11 correct. The paper's main claim is that the dual-agent reasoner/critic loop significantly improves query generation reliability.
Significance. If the reported accuracy is reproducible, the paper demonstrates that open-weight LLMs combined with a hand-built schema and an iterative reasoner/critic loop can answer realistic ERP queries, which is practically relevant given data-privacy constraints. The authors provide a concrete recipe and honestly describe several deployment challenges, including the need for a manual semantic schema, vague prompts, and intermittent loss of user intent. However, because the central reliability claim is not tested against any single-agent or reasoner-only baseline, and because the evaluation rests on a small, unreleased, single-expert test set, the significance of the claimed architecture-level contribution is not yet established.
major comments (3)
- [§3.2, Table 3] The central claim that the dual-agent architecture "significantly improves" query reliability is not supported by the reported experiments, because every result in Table 3 comes from the complete pipeline (REACT, semantic schema, execution feedback, critic loop, HITL, and structured extraction). There is no condition with the critic disabled, no single-pass reasoner-only baseline, and no comparison against direct SQL generation from the same schema and prompts. Since the paper's novelty claim is specifically about the reasoning-plus-critique loop, an ablation isolating that component is necessary.
- [§4.2 and §5] The evaluation set is described as twelve queries in §4.2, but the results section and Table 3 report eleven questions, and §4.3 says eleven models were employed while Table 3 lists only nine scored models. The authors should reconcile these counts and report whether the two models that exceeded the generation time limit (Qwen 3 32B Q4 and Llama 3.3 70B Q4) are counted as evaluated models; as written, the reader cannot determine the effective sample size.
- [§4.2 and §5] The reliability measurement rests entirely on a hand-curated set of at most twelve queries written by an experienced ERP user and judged by a single expert, with no release of the query set, no scoring rubric, and no inter-rater reliability check. Because the paper claims a general recipe, this is a load-bearing weakness; the authors should either make the queries and judging criteria available (in anonymized form) or provide a second annotator and a more detailed rubric.
minor comments (5)
- [Throughout] There are typographical issues that should be corrected, including "thethe" in §3.2, "T able" in §3.3 and Table 1, "Additionaly" in §4.3, "an the underlying data" in §6, "F uture W ork" in §7, and a double period in §7 ("plug-and-play agent..").
- [§4.3] The statement that prompts were built around Qwen 2.5 32B is an important caveat, but the paper merely says a disadvantage is expected; a prompt-sensitivity analysis would strengthen the evaluation and help readers understand how model-specific the recipe is.
- [§2] The related-work discussion cites several multi-agent text-to-SQL systems, such as MAG-SQL and SQLFixAgent, but does not compare the proposed architecture's components with those existing designs; a brief comparison table or a more explicit positioning of the contribution would help.
- [§5] The accuracy metric is simply the number of correct answers; since some queries may be more complex than others, reporting per-query difficulty or at least the distribution of error types would improve interpretability.
- [§1] The abstract claims "significantly improving" but the results section does not report any statistical test; the authors should either moderate the language or provide evidence of significance.
Circularity Check
No significant circularity found: the paper's empirical claims are under-supported by missing baselines, but the reported results are not constructed from their inputs.
full rationale
The paper contains no formal derivation, fitted parameters, or equations, so the classic circularity patterns (self-definitional reduction, fitted input called prediction, self-citation chains, ansatz smuggled via citation, renaming known results) do not apply. The architecture's components (REACT agent, reasoner/critic loop, handcrafted Markdown schema, HITL clarification) are stated as design choices, and the evaluation is an expert-validated accuracy count on 11/12 curated questions. The expert-written semantic schema is an input to the agent, but the expert validation is an independent judgment of query correctness, not a quantity derived from the schema. The paper's own Section 7 acknowledges that the agent requires expert-written documentation and that vague prompts cause column-selection failures, which is a scalability limitation, not a circular step. The most serious weakness is that no ablation or single-pass baseline isolates the contribution of the dual-agent loop, so the claim that the structure 'significantly impro[es] their accuracy' (Section 3.2) is not demonstrated; however, absence of evidence is a correctness and internal-validity problem, not circularity. Similarly, prompt tuning around Qwen 2.5 32B (Section 4.3) may bias model comparisons but does not render the reported 10/11 and 9/11 scores equivalent to the input prompts by construction. The test-set count inconsistency (12 in Section 4.2 vs 11 in Section 5) is a reporting defect, not circularity. No self-citations are load-bearing; all cited works (Chen et al. 2023, Xie et al. 2024, Cen et al. 2025) are external. Therefore the circularity score is 0.
Assumptions & free parameters
free parameters (4)
- Max reasoner attempts N
- Max critic rounds M
- Prompt templates =
Tuned for Qwen 2.5 32B
- Generation time limit =
3 minutes
assumptions (4)
- domain assumption An expert-written natural-language description of the ERP schema is necessary and sufficient for the LLM to map user intents to the correct columns.
- domain assumption The expert-curated questions and the expert's binary ✓/✗ judgments are a correct and representative measure of the agent's performance.
- domain assumption Open-weight LLMs can interpret SQL error messages and refine their own SQL accordingly.
- domain assumption The 7-table, 321-column anonymized ERP schema is representative enough to generalize the recipe to other production ERP databases.
Cite this review
Pith. "Pith review of Chatting with your ERP: A Recipe." pith.science (2026). https://pith.science/paper/F5UHVILG
@misc{pith2026250723429,
author = {Pith},
title = {Pith review of: Chatting with your ERP: A Recipe},
year = {2026},
howpublished = {\url{https://pith.science/paper/F5UHVILG}},
note = {Machine review of arXiv:2507.23429}
}
read the original abstract
This paper presents the design, implementation, and evaluation behind a Large Language Model (LLM) agent that chats with an industrial production-grade ERP system. The agent is capable of interpreting natural language queries and translating them into executable SQL statements, leveraging open-weight LLMs. A novel dual-agent architecture combining reasoning and critique stages was proposed to improve query generation reliability.
Reference graph
Works this paper leans on
-
[1]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in ":" * " " * FUNCTION f...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize "" * " " * ...
-
[3]
author Cen, J. , author Liu, J. , author Li, Z. , & author Wang, J. ( year 2025 ). title Sqlfixagent: Towards semantic-accurate text-to-sql parsing via consistency-enhanced multi-agent collaboration . https://arxiv.org/abs/2406.13408. arXiv:2406.13408 http://arxiv.org/abs/2406.13408
arXiv 2025
-
[4]
author Chen, X. , author Lin, M. , author Schärli, N. , & author Zhou, D. ( year 2023 ). title Teaching large language models to self-debug . https://arxiv.org/abs/2304.05128. arXiv:2304.05128 http://arxiv.org/abs/2304.05128
arXiv 2023
-
[5]
author Gao, D. , author Wang, H. , author Li, Y. , author Sun, X. , author Qian, Y. , author Ding, B. , & author Zhang, J. ( year 2023 ). title Text-to-sql empowered by large language models: A benchmark evaluation . https://arxiv.org/abs/2308.15363. arXiv:2308.15363 http://arxiv.org/abs/2308.15363
arXiv 2023
-
[6]
author Hochreiter, S. , & author Schmidhuber, J. ( year 1997 ). title Long short-term memory . journal Neural Computation \/ , volume 9 \/ , pages 1735--1780 . https://doi.org/10.1162/neco.1997.9.8.1735. :10.1162/neco.1997.9.8.1735. arXiv:https://direct.mit.edu/neco/article-pdf/9/8/1735/813796/neco.1997.9.8.1735.pdf http://arxiv.org/abs/https://direct.mit...
-
[7]
author Li, F. , & author Jagadish, H. V. ( year 2014 ). title Constructing an interactive natural language interface for relational databases . journal Proc. VLDB Endow. \/ , volume 8 \/ , pages 73–84 . https://doi.org/10.14778/2735461.2735468. :10.14778/2735461.2735468
arXiv 2014
-
[8]
author Liu, A. , author Hu, X. , author Wen, L. , & author Yu, P. S. ( year 2023 ). title A comprehensive evaluation of chatgpt's zero-shot text-to-sql capability . https://arxiv.org/abs/2303.13547. arXiv:2303.13547 http://arxiv.org/abs/2303.13547
arXiv 2023
Show all 10 references
-
[9]
, author Li, R
author Rajkumar, N. , author Li, R. , & author Bahdanau, D. ( year 2022 ). title Evaluating the text-to-sql capabilities of large language models . https://arxiv.org/abs/2204.00498. arXiv:2204.00498 http://arxiv.org/abs/2204.00498
2022 arXiv
-
[10]
, author Wu, G
author Xie, W. , author Wu, G. , & author Zhou, B. ( year 2024 ). title Mag-sql: Multi-agent generative approach with soft schema linking and iterative sub-sql refinement for text-to-sql . https://arxiv.org/abs/2408.07930. arXiv:2408.07930 http://arxiv.org/abs/2408.07930
2024 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.