Pith. sign in

REVIEW 4 major objections 5 minor 25 references

Contrato360 2.0: A Document and Database-Driven Question-Answer System using Large Language Models and Agents

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Router agent answers contract questions from PDFs and databases

desk verdict Useful applied system with a real chunking idea, but the headline improvement claim is not supported by the paper's own evaluation; deserves revision, not rejection. read the letter →

arxiv 2412.17942 v1 pith:MJELNTFZ submitted 2024-12-23 cs.AI

Antony Seabra , Claudio Cavalcante , Joao Nepomuceno , Lucas Lago , Nicolaas Ruberg , Sergio Lifschitz This is my paper · ORCID
classification cs.AI
keywords questionansweringcontractmanagementretrieval-augmentedgenerationtext-to-SQLLLMagentspromptengineeringvectordatabasesenterpriseinformationsystems
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper presents Contrato360 2.0, a question-answering system for contract management that draws on two sources at once: the text of contract PDFs and the structured records of a contract management database. The authors claim that routing every question through a router agent to a retrieval-augmented generation (RAG) agent and a text-to-SQL agent, then assembling the answer with prompt-engineered instructions, makes answers more relevant and accurate than using any single technique on its own, and does so without retraining the language model. They report that on a benchmark of prepared questions, specialists rated 59 of 60 direct questions correct and 65 of 80 indirect questions correct, with the remainder incomplete. A sympathetic reader would take the paper's contribution to be an architecture showing that enterprise document-and-database Q&A can be assembled from orchestrated components rather than fine-tuned models.

What carries the argument

The load-bearing mechanism is the multi-agent orchestration workflow, in particular the router agent that acts as the decision-maker for every query. In parallel, a RAG agent retrieves document chunks from a vector store and a SQL agent retrieves structured records from the contract database; the retrieved pieces are then combined by prompt engineering into the final answer. A supporting mechanism is metadata-enriched chunking: contracts are split by clause rather than by fixed token count, and each chunk carries the source file, the OCS contract number, and the clause title. The metadata filter is what prevents querying one contract from returning a textually similar clause of a different contract, addressing the paper's stated similarity-versus-relevance problem in standardized legal documents. The SQL agent is constrained to read-only queries, which the authors present as the safety design of the system.

What would settle it

Run the same direct and indirect benchmark questions against Contrato360 1.0, against a RAG-only configuration, and against a SQL-only configuration, with raters blind to which version produced each answer. If the full agent system does not achieve a higher correct rate than the baselines on the same question set, the claim of significant improvement fails. A simpler check: recruit raters who did not help write the questions and see whether the indirect-question scores, especially the waiver-of-bidding and exemption-from-tender items, stay at the reported level.

Watch

Extended reading notes

Core claim

On its own terms, the paper's claim is that a question-answering system can be made substantially more useful for contract managers by replacing a single static pipeline with four coordinated techniques: section-based chunking of contract PDFs enriched with metadata (source file, contract number, clause), retrieval-augmented generation over the resulting vector store, a text-to-SQL agent that maps natural-language questions onto the contract database schema, and prompt engineering that fixes response style and forces answers to cite the contract number. The router agent decides whether a question belongs to the contract domain and then sends in-domain questions to the RAG and SQL agents in parallel; when the retrieved material includes tabular values, a graph agent can add a visualization. The authors' reported evaluation is a benchmark of prepared direct and indirect questions judged by two domain specialists, and their conclusion is that the agent orchestration combines both data sources well enough to answer most questions correctly without any fine-tuning of the language model. The paper frames this as a general direction for information systems: compose RAG, text-to-SQL, agents, and prompt engineering rather than retraining a model for each new document corpus.

Load-bearing premise

The claim of significantly improved accuracy rests on a benchmark in which two specialists from the deploying organization rated the system's answers to questions the authors prepared, with no comparison against the earlier version of the system or against a configuration missing one of the four techniques; if those ratings are not representative, the central claim is unsupported.

Editorial extensions

If this is right

  • Contract managers can ask questions that span a PDF and the database, such as the contract manager plus the penalty clause, and receive one consolidated answer instead of searching documents by hand.
  • New contracts can be added to the system by chunking and embedding their PDFs and loading their records into the database; the language model itself does not need to be retrained for the new content.
  • Because text-to-SQL handles the structured half of the query, users do not need to know SQL or the database schema to get exact figures from the contract system.
  • The router's domain filter gives the system a bounded scope, so open-domain questions are deflected rather than answered from the model's general knowledge.
  • The same composition of RAG, text-to-SQL, agents, and prompt engineering is presented as transferable to other document-plus-database enterprise domains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's accuracy conclusion is not yet supported by a comparison: the benchmark lacks a run against the earlier Contrato360 version or against configurations with individual techniques removed, so the reader cannot tell which component contributes the gain.
  • The metadata-filtering fix suggests a direct ablation test: removing the contract-number metadata from chunks should measurably lower retrieval precision on cross-contract questions such as 'what is the object of contract OCS 278/2023'.
  • The router's parallel split between RAG and SQL paths raises a boundary question: for a query that needs both a database figure and a document clause, the system retrieves both in parallel and merges them in the prompt, so the merge step is where a wrong answer would hide.
  • The architecture generalizes to any regulated document-heavy workflow where the same entity appears in both free-text documents and structured records; the cost is engineering per-domain metadata and schema, not model training.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents Contrato360 2.0, a question-answering system for contract management that combines Retrieval-Augmented Generation over contract PDFs, a text-to-SQL agent over a contract management database, prompt engineering, and router/graph agents orchestrated by an LLM. The system uses gpt-4-turbo for answer generation and OpenAI embeddings, with no model retraining. The authors report a pilot evaluation with 75 contracts and two BNDES IT specialists, classifying answers as Correct or Incomplete for both direct questions (answered from PDFs) and indirect questions (answered from database data), and they claim that the multi-agent orchestration and combination of techniques significantly improve answer relevance and accuracy.

Significance. If properly validated, the system would provide a useful industrial demonstration of LLM-based agents for enterprise contract management, with practical time savings and integration of unstructured documents with structured databases. The paper's strengths are its concrete architecture, the use of metadata-augmented chunking to address the similarity-versus-relevance problem in standardized contracts, SQL-command safety validation, and a real-world pilot with a Brazilian public bank. However, the current evaluation does not support the abstract's strong causal claim: there is no baseline, no ablation, no statistical analysis, and the reported results include a clear failure on an indirect-question class. The contribution is therefore best regarded as an engineering report whose central claim requires substantially stronger evidence.

major comments (4)
  1. [Abstract and Section 5] The central claim that the multi-agent orchestration and combination of techniques 'significantly improve the relevance and accuracy of the answers' is not supported by the reported evaluation. Section 5 contains no comparison against Contrato360 1.0, no ablation in which agents, RAG, text-to-SQL, or prompt engineering are individually removed, and no statistical test or confidence measure. The word 'significantly' is used as a rhetorical claim rather than a measured one; the authors should either add a controlled comparison or replace this claim with a qualified statement about observed performance in a pilot setting.
  2. [Section 5, Tables 1 and 2] The evaluation's measurement unit is unclear and the rater setup is under-specified. Each row in Tables 1 and 2 shows counts that sum to 10, but the text states that only two BNDES IT specialists were involved; it is not explained whether each rater answered each question five times, whether questions were re-run across sessions, or how the aggregate counts were produced. The raters are also not described as blinded to the system design or to the intended answer, and no inter-rater agreement is reported. Without this information, the 10-item accuracy counts cannot be interpreted as a reliable evaluation.
  3. [Section 5, Table 2] The worst reported result is a direct counterexample to the claimed improvement for indirect questions: 'How many DLs (Exemptions from Tenders) were contracted in yy?' received 0 Correct and 10 Incomplete, and the authors attribute this to the 'Waiver of Bidding' concept not being well captured. The paper says that an adjustment in queries and/or prompts could add this semantics, but no follow-up experiment is shown. Because indirect questions over database records are a stated core contribution, this failure needs to be analyzed and addressed, not deferred.
  4. [Section 5, Table 1] The statement that the system presents relevant answers for direct questions 'for all experiments' is contradicted by Table 1, row 2, which shows 9 Correct and 1 Incomplete for the question about contract subject. The authors should either correct this statement or explain why an 'Incomplete' answer still counts as 'relevant'. A reader cannot reconcile the summary sentence with the table as printed.
minor comments (5)
  1. [Section 2.2] There is a typo in the distance metric list: 'Manhatan' should be 'Manhattan'. Please also check the spelling of 'Euclidean' and ensure the metric names are set in a consistent style.
  2. [Section 3.4] There are several language errors, including 'understanting' for 'understanding' and the sentence fragment 'In A question on the contract domain will follow our workflow to find a relevant answer.' The paragraph should be rewritten for clarity.
  3. [Section 6] The sentence 'The 8 demonstrates the ability of Contrato360 in' should read 'Figure 8 demonstrates the ability of Contrato360 to...'. More generally, every figure should be explicitly referenced in the text at the point where it is discussed.
  4. [Section 5] The footnote states that a third category 'Incorrect' was not obtained, but an 'Incomplete' answer is not necessarily a fully relevant answer. The binary Correct/Incomplete scheme should be justified, and the paper should clarify whether 'Incomplete' answers still count as successes or as partial failures in the reported claims.
  5. [References] The LangChain documentation URL is broken across lines and should be given as a single, clickable link. Several citations are also formatted inconsistently (e.g., '(Li et al., 2022)' and '(Yao et al., 2023)'); please harmonize the reference formatting.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the system is described and evaluated behaviorally; the self-citations are background, and the evaluation weakness is an evidential gap, not a derivation that reduces to its own inputs.

full rationale

This paper contains no formal derivation chain whose conclusion is equivalent to its premises. It describes a Q&A system built from RAG, text-to-SQL, agents, and prompt engineering, and reports a human evaluation in Section 5. No quantity is defined in terms of another, no parameter is fitted and then reported as a prediction, and no uniqueness theorem is invoked to force a choice. The only self-citations are background references to the authors' prior Contrato360 work (e.g., 'To enhance the relevance of user queries, prior work by (Seabra et al., 2024) employed Retrieval-Augmented Generation (RAG) techniques and a static approach to text-to-SQL') and to a Webist paper co-authored by one of the present authors; these do not carry the load of the paper's central claim. The evaluation's weaknesses are real but evidentiary: two BNDES specialists, no baseline comparison, no statistical tests, and a 0/10 result on one indirect question. Those weaknesses undermine the 'significantly improve' claim as a matter of validation, not circularity. There is no construction by which the reported Correct/Incomplete categories are forced by the system's design, nor is any predicted quantity algebraically identical to an input. Accordingly, the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The system's claims rest on unvalidated assumptions: that GPT-4 generates correct SQL from schema descriptions, that embedding similarity reflects semantic relevance after metadata filtering, that clause-based chunking preserves answerable units, and that the two-specialist evaluation measures true answer quality. None of these are tested against external benchmarks or released artifacts.

assumptions (4)
  • domain assumption GPT-4-turbo can generate correct SQL queries from natural language given the database schema and entity recognition.
    Section 3.2 states the SQL agent performs entity recognition, maps entities to tables and columns, and prepares SQL, but no SQL correctness evaluation is reported.
  • domain assumption Embedding similarity correlates with semantic relevance for contract sections, especially after metadata filtering.
    Section 3.1 discusses the similarity/relevance problem and proposes metadata filtering, but no retrieval quality metrics (e.g., recall@k) are provided.
  • domain assumption Chunking by contract section preserves semantic units sufficient for answering queries.
    Section 3.1 describes clause-based chunking but does not compare it to token-based chunking or other alternatives.
  • domain assumption The two BNDES specialists' ratings are a valid measure of answer quality.
    Section 5 describes an evaluation by two specialists from the authors' organization using author-prepared questions, with no inter-rater reliability or blinded protocol.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Contrato360 2.0: A Document and Database-Driven Question-Answer System using Large Language Models and Agents." pith.science (2026). https://pith.science/paper/MJELNTFZ

@misc{pith2026241217942,
  author       = {Pith},
  title        = {Pith review of: Contrato360 2.0: A Document and Database-Driven Question-Answer System using Large Language Models and Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MJELNTFZ}},
  note         = {Machine review of arXiv:2412.17942}
}
read the original abstract

We present a question-and-answer (Q\&A) application designed to support the contract management process by leveraging combined information from contract documents (PDFs) and data retrieved from contract management systems (database). This data is processed by a large language model (LLM) to provide precise and relevant answers. The accuracy of these responses is further enhanced through the use of Retrieval-Augmented Generation (RAG), text-to-SQL techniques, and agents that dynamically orchestrate the workflow. These techniques eliminate the need to retrain the language model. Additionally, we employed Prompt Engineering to fine-tune the focus of responses. Our findings demonstrate that this multi-agent orchestration and combination of techniques significantly improve the relevance and accuracy of the answers, offering a promising direction for future information systems.

Figures

Figures reproduced from arXiv: 2412.17942 by the authors.

Figure 1
Figure 1. Retrieval-Augmented Generation mize the potential of LLMs by providing them with instructions and context (OpenAI, 2023b). In the context of prompt engineering, prompts are a fundamental part of the process. Through prompts, engineers can outline the script for a re￾sponse, specifying the desired style and format for the LLM response (White et al., 2023) (Giray, 2023). For example, to define the style of a conversat… view at source ↗
Figure 2
Figure 2. Methodology Workflow Combining Different Techniques [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Chunking applied to Contracts [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Contracts metadata agers. The LangChain SQL Agent (Langchain, 2024) has proven to be a highly flexible tool for interact￾ing with the CMS database. Upon system startup, our SQL agent establishes an authenticated connec￾tion to the database and retrieves the schema. Whe…
Figure 5
Figure 5. Figure 5: Application architecture nents. At the heart of the system lies the Backend Layer, which is primarily managed by the Backend Agents (Python and Langchain). This layer orchestrates in￾teractions between the document processing, vector storage, contracts database, and th…
Figure 6
Figure 6. Figure 6: Contracts Q&A Streamlit application [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Plotly Agent ployed four techniques to improve the relevance of the answers: 1) Augmented Retrieval (RAG) combined with semantic augmentation using metadata to re￾trieve information from PDFs; 2) Text-to-SQL, aggre￾gating dynamic information from the contracts made ava…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 11 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...

  2. [2]

    Chen, J., Lin, H., Han, X., and Sun, L. (2024). Benchmarking large language models in retrieval-augmented generation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 17754--17762

    work page 2024

  3. [3]

    Fan, W., Ding, Y., Ning, L., Wang, S., Li, H., Yin, D., Chua, T.-S., and Li, Q. (2024). A survey on rag meeting llms: Towards retrieval-augmented large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 6491--6501

    work page 2024

  4. [4]

    Feng, Z., Feng, X., Zhao, D., Yang, M., and Qin, B. (2024). Retrieval-generation synergy augmented large language models. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 11661--11665. IEEE

    work page 2024

  5. [5]

    Gao, D., Wang, H., Li, Y., Sun, X., Qian, Y., Ding, B., and Zhou, J. (2023a). Text-to-sql empowered by large language models: A benchmark evaluation. arXiv preprint arXiv:2308.15363

    arXiv 2023

  6. [6]

    Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., and Wang, H. (2023b). Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997

    arXiv 2023

  7. [7]

    Giray, L. (2023). Prompt engineering with chatgpt: a guide for academic writers. Annals of biomedical engineering , 51(12):2629--2633

    work page 2023

  8. [8]

    Jeong, C. (2023). A study on the implementation of generative ai services using an enterprise data-based llm application architecture. arXiv preprint arXiv:2309.01105

    arXiv 2023

Show all 25 references
  1. [9]

    Jin, H., Huang, L., Cai, H., Yan, J., Li, B., and Chen, H. (2024). From llms to llm-based agents for software engineering: A survey of current, challenges and future. arXiv preprint arXiv:2408.02479

    2024 arXiv

  2. [10]

    Langchain agents documentation

    Langchain (2024). Langchain agents documentation. https://python.langchain.com/v0.1/docs/use_cases/sql/agents/. Accessed: 2024-09-06

    2024

  3. [11]

    u ttler, H., Lewis, M., Yih, W.-t., Rockt \

    Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., K \"u ttler, H., Lewis, M., Yih, W.-t., Rockt \"a schel, T., et al. (2020). Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems , 33:9459--9474

    2020

  4. [12]

    Li, H., Su, Y., Cai, D., Wang, Y., and Liu, L. (2022). A survey on retrieval-augmented text generation. arXiv preprint arXiv:2202.01110

    2022 arXiv

  5. [13]

    Liu, A., Hu, X., Wen, L., and Yu, P. S. (2023). A comprehensive evaluation of chatgpt's zero-shot text-to-sql capability. arXiv preprint arXiv:2303.13547

    2023 arXiv

  6. [14]

    Mialon, G., Dess \` , R., Lomeli, M., Nalmpantis, C., Pasunuru, R., Raileanu, R., Rozi \`e re, B., Schick, T., Dwivedi-Yu, J., Celikyilmaz, A., et al. (2023). Augmented language models: a survey. arXiv preprint arXiv:2302.07842

    2023 arXiv

  7. [15]

    Chatgpt fine-tune description

    OpenAI (2023a). Chatgpt fine-tune description. https://help.openai.com/en/articles/6783457-what-is-chatgpt. Accessed: 2024-03-01

    2023

  8. [16]

    Chatgpt prompt engineering

    OpenAI (2023b). Chatgpt prompt engineering. https://platform.openai.com/docs/guides/prompt-engineering. Accessed: 2024-04-01

    2023

  9. [17]

    Pinheiro, J., Victorio, W., Nascimento, E., Seabra, A., Izquierdo, Y., Garc a, G., Coelho, G., Lemos, M., Leme, L. A. P. P., Furtado, A., et al. (2023). On the construction of database interfaces based on large language models. In Proceedings of the 19th International Conferen...

    2023

  10. [18]

    Saeed, M., De Cao, N., and Papotti, P. (2023). Querying large language models with sql. arXiv preprint arXiv:2304.00472

    2023 arXiv

  11. [19]

    Seabra, A., Nepomuceno, J., Lago, L., Ruberg, N., and Lifschitz, S. (2024). Contrato360: uma aplicação de perguntas e respostas usando modelos de linguagem, documentos e bancos de dados. In Anais do XXXIX Simpósio Brasileiro de Bancos de Dados

    2024

  12. [20]

    Singh, A., Ehtesham, A., Kumar, S., and Khoei, T. T. (2024). Enhancing ai systems with agentic workflows patterns in large language model. In 2024 IEEE World AI IoT Congress (AIIoT) , pages 527--532. IEEE

    2024

  13. [21]

    N., Kaiser, ., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, ., and Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems , 30

    2017

  14. [22]

    Wang, M., Wang, M., Xu, X., Yang, L., Cai, D., and Yin, M. (2023). Unleashing chatgpt's power: A case study on optimizing information retrieval in flipped classrooms via prompt engineering. IEEE Transactions on Learning Technologies

    2023

  15. [23]

    S., Mishra, S., Perot, V., Zhang, Y., Mattapalli, A., Taly, A., Shang, J., et al

    Wang, Z., Wang, Z., Le, L., Zheng, H. S., Mishra, S., Perot, V., Zhang, Y., Mattapalli, A., Taly, A., Shang, J., et al. (2024). Speculative rag: Enhancing retrieval augmented generation through drafting. arXiv preprint arXiv:2407.08223

    2024 arXiv

  16. [24]

    White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., Elnashar, A., Spencer-Smith, J., and Schmidt, D. C. (2023). A prompt pattern catalog to enhance prompt engineering with chatgpt. arXiv preprint arXiv:2302.11382

    2023 arXiv

  17. [25]

    Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. (2023). React: Synergizing reasoning and actin in langage models. arXiv preprint arXiv:2210.03629v3

    2023 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.