Dynamic Multi-Agent Orchestration and Retrieval for Multi-Source Question-Answer Systems using Large Language Models

Antony Seabra; Claudio Cavalcante; Joao Nepomuceno; Lucas Lago; Nicolaas Ruberg; Sergio Lifschitz

REVIEW 4 major objections 4 minor 13 references

Dynamic Multi-Agent Orchestration and Retrieval for Multi-Source Question-Answer Systems using Large Language Models

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A router agent that dispatches each query to a RAG-based document agent or a text-to-SQL database agent, with prompts adapted per agent, yields more accurate and relevant answers in a multi-source contract-management question-answer system.

desk verdict A readable engineering write-up of an enterprise contract-QA system, but the central accuracy claim rests on two specialists' qualitative feedback with no baseline; not a research result as it stands. read the letter →

arxiv 2412.17964 v1 pith:Z4GYQQY3 submitted 2024-12-23 cs.AI

Antony Seabra , Claudio Cavalcante , Joao Nepomuceno , Lucas Lago , Nicolaas Ruberg , Sergio Lifschitz This is my paper · ORCID

classification cs.AI

keywords InformationRetrievalQuestionAnswerLargeLanguageModelsDocumentsDatabasesPromptEngineeringAugmentedGenerationText-to-SQL

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a question-answering architecture that combines retrieval-augmented generation, text-to-SQL, and dynamically adapted prompts, coordinated by specialized agents. Its central claim is that routing each query to the right agent—RAG for unstructured contract text, SQL for structured database values—produces more accurate and relevant answers than a single generic retrieval pipeline. The system is demonstrated in contract management, where questions often span both PDF documents and relational records. The paper's evidence is qualitative feedback from two domain specialists on a set of 75 contracts, arguing the approach saves time and yields complete answers for direct questions, with some incompleteness on semantically tricky indirect queries.

What carries the argument

The carrying mechanism is the agent-based orchestration pipeline: a Router Agent uses predefined regular expressions to identify query patterns and dispatch each question to the appropriate specialist agent. The RAG Agent retrieves semantically similar chunks from a vector database, filtered by metadata such as source, contract, and clause, to resolve the similar-but-irrelevant ambiguity common in contracts; the SQL Agent translates natural-language questions into exact SQL over a relational database; and a Graph Agent prompts the model to render tabular results as bar graphs. Dynamic prompt engineering tailors the instructions to whichever agent handles the query, adapting the LLM's behavior without retraining.

What would settle it

Collect a held-out set of user queries whose wording intentionally avoids the router's known keyword and regex patterns (synonyms, paraphrases, indirect phrasings), run them through the system, and compare the fraction of wrong-agent routes and incorrect answers against a single-pipeline RAG baseline; if routing errors are common and responses degrade accordingly, the central claim fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that a multi-agent orchestration layer, built on a Router Agent that classifies queries by regular expression patterns, can dynamically select between a RAG Agent for unstructured document retrieval via vector embeddings and metadata filtering and a SQL Agent for exact structured querying via text-to-SQL, with dynamic prompts per agent improving response accuracy and contextual relevance. In the contract-management testbed, the authors report that direct questions—those answerable from contract PDFs and their metadata—were consistently answered completely, while indirect questions requiring database values were generally satisfactory, with incomplete answers for terms like 'Waiver of Bidding' traced to routing or semantic retrieval gaps.

Load-bearing premise

The Router Agent's predefined regular expressions must correctly classify every user query into the correct retrieval path; a query phrased outside these patterns gets sent to the wrong agent and the claimed accuracy advantage collapses.

Editorial extensions

If this is right

A system built on this methodology answers both document-content questions and exact-value database questions in one interface, removing the need to search PDFs manually.
Because prompts adapt per agent, the same LLM can be steered for interpretive text synthesis, precise SQL generation, and graphical summaries without retraining.
Metadata filtering on chunks (source, contract, clause) reduces the risk of retrieving semantically similar but irrelevant contract sections.
The modular agent design supports adding new retrieval paths, such as graph or API agents, without rearchitecting the system.
Indirect questions with complex legal semantics, such as 'Waiver of Bidding,' remain a known weak point, and the authors expect prompt or query-structure adjustments to improve them.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The qualitative feedback suggests perceived gains, but a quantitative benchmark comparing the multi-agent system against single-pipeline RAG and text-to-SQL baselines on the same query set would directly test and sharpen the claimed improvement.
The regex-based router is the natural failure point; replacing the hand-written patterns with a trained classifier, or with the LLM itself making the routing decision, could make the approach robust to paraphrase and scale to new domains.
The same orchestration pattern should transfer to other mixed-source domains, such as medical records (clinical notes plus structured lab data) or finance (prospectus PDFs plus market databases), where the metadata-chunk filtering idea carries over.
Embedding structured data as text into the vectorstore and querying it via SQL are treated as alternatives; a hybrid that first routes and then fuses results from both paths could handle queries that need both exact values and interpretive context.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Quick take: this is a straightforward system description of Contrato360, a multi-agent QA pipeline that combines RAG, text-to-SQL, dynamic prompts, and a regex-based router for contract management at BNDES. If you treat it as an engineering case study, it's a decent one. If you take the abstract at its word—that the methodology 'enhances response accuracy and relevance'—the paper doesn't earn that claim.

What it does well: the writing is clear and honest. The authors explain why naive RAG fails on contract documents (similar-sounding clauses that don't answer the query) and how they use metadata filtering over contract number and clause to narrow retrieval. That's a practical, sensible fix that many practitioners will recognize. The per-agent dynamic prompts and the Plotly graph agent are small but useful additions. The paper also states its limitations openly: the router uses predefined regular expressions, and some indirect questions (e.g., 'Waiver of Bidding') returned incomplete answers.

The soft spots are real and central. Section 4 is the entire evaluation: two IT contract specialists from the authors' own organization, a set of 75 contracts, and no quantitative metrics, no baseline system, no inter-rater agreement, no statistical test. The phrase 'results indicate that this approach enhances response accuracy and relevance' has no comparator. A single retrieval pipeline, or even a non-agent RAG baseline, would be the obvious thing to test against. The router's regex reliance is another load-bearing assumption; the paper concedes it as future work, but if the regexes fail, the whole routing decision misfires and the accuracy gain collapses. On novelty, the core system is prior work by the same group, and the component techniques are all cited from the literature. The incremental pieces are variations on established agentic patterns.

Who is this for? Practitioners building enterprise QA over mixed PDF/database sources, and readers who want a candid account of a real deployment. It is not a research result that advances the field. I would not send this to a top-tier venue; desk reject is defensible. If the authors add a proper benchmark with public data, baselines, and metrics, it could become a useful application paper.

Referee Report

4 major / 4 minor

Summary. The paper proposes a multi-agent orchestration and retrieval methodology for multi-source question-answer systems, combining retrieval-augmented generation (RAG) for unstructured PDF documents, text-to-SQL for structured databases, dynamic prompt engineering, and a router agent that uses predefined regular expressions to select the retrieval path. The methodology is instantiated in Contrato360, a contract-management QA system, and is evaluated through qualitative feedback from two IT contract specialists from BNDES on a set of 75 contracts with direct and indirect questions. The authors claim that the approach enhances response accuracy and relevance compared with traditional single-pipeline systems.

Significance. If the evaluation were rigorous, the paper would provide a useful case study of integrating RAG, text-to-SQL, and agent-based routing in a real-world contract-management domain. The system design, especially the metadata-filtered section-based chunking, is plausible and may be of practical interest to practitioners. However, as presented, the contribution is primarily an engineering integration of known components: there are no new algorithms, no machine-checked proofs, no reproducible benchmark, and no quantitative evaluation. The evidence for the headline claim of enhanced accuracy and relevance is anecdotal and does not yet meet the standard of a journal paper.

major comments (4)

[Section 4 (Evaluation)] The abstract claims that the proposed approach 'enhances response accuracy and relevance,' but Section 4 reports only qualitative feedback from two IT contract specialists, with no baseline system, no quantitative accuracy or relevance metrics, no inter-rater agreement, and no statistical test. Because the central claim is comparative ('enhances'), a comparator is required. Please add a controlled comparison against at least one baseline pipeline (e.g., RAG-only or SQL-only), define and measure accuracy/relevance metrics (e.g., exact match, F1, or a rubric-based score), and report evaluator agreement or a structured scoring procedure. Without this, the claim is unsupported by the presented evidence.
[Section 3.3 and Section 5] The Router Agent is a load-bearing component of the architecture: it uses predefined regular expressions to decide between the RAG and SQL paths, and Section 5 concedes that the router 'relies on predefined regular expressions to route queries' and proposes machine learning as future work. If a query does not match the patterns, the wrong agent is selected and the system's accuracy degrades. The paper provides no data on routing accuracy, pattern coverage, or failure cases, and no fallback mechanism is described. Please report the router's precision and recall on the benchmark questions, and describe how out-of-pattern queries are handled in practice.
[Section 3.1 (Embeddings models)] The text states that the authors selected 'text-davinci-002, a model from OpenAI's GPT-3.5 family, along with embeddings with 1536 dimensions.' text-davinci-002 is a completion model, not an embedding model; the configuration as described is not reproducible. Please identify the actual embedding model used (e.g., text-embedding-ada-002), its dimensionality, and the embedding configuration, or clarify whether the system used a separate embedding model for the vectorstore.
[Section 4 (Results)] The evaluation reports that indirect questions such as those involving 'Waiver of Bidding' produced incomplete answers, yet the abstract and conclusion make an unqualified claim of enhanced accuracy and relevance. Please either temper the claim to match the evidence or provide a detailed error analysis and demonstrate that the proposed prompt adjustments resolve the incompleteness. As written, the reported limitations are not reconciled with the positive conclusion.

minor comments (4)

[General] The manuscript contains several typos and incomplete placeholders: 'Figure ??' appears in Sections 3.1 and 3.4, and 'Manhatan' (Section 3.1), 'the the' (Section 3.3), 'lenghty' (Section 3.1), 'prompt'' (Introduction), and 'V ectorstore' (Section 3.1) should be corrected.
[Section 3.1] The RAG process is described in seven steps, but Figure 1 is not displayed in the manuscript; please either include the figure or remove the reference to it.
[Section 4] The paper refers to several figures (e.g., Figures 5–8) that appear to be screenshots without detailed captions; please add explanatory captions and ensure all figures are actually included and sequentially numbered.
[References] The reference list contains inconsistencies (e.g., 'Gao et al., 2023a' and 'Gao et al., 2023b' are interleaved; 'OpenAI, 2023a' cites a help page rather than a citable publication). Please standardize the bibliography.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is a descriptive methodology and qualitative evaluation; the self-citation to Contrato360 is contextual, not load-bearing.

full rationale

The paper contains no mathematical derivation, no fitted parameters, and no quantity defined in terms of another. The central claim that the orchestration 'enhances response accuracy and relevance' is an empirical assertion supported by Section 4's qualitative specialist feedback, not by a reduction of outputs to inputs. The self-citation to [Seabra et al., 2024] appears in background statements ('According to [Seabra et al., 2024], the first step when applying RAG involves...') and in the statement that the methodology was implemented in Contrato360; this is descriptive context and is not the evidence for the claimed enhancement. Section 5's concession that the Router Agent 'relies on predefined regular expressions' is an acknowledged limitation, and Section 4's lack of a baseline and metrics is a soundness and selection-bias concern, not a circularity concern. No equation, definition, or self-citation chain forces the claimed result to be true by construction, so no circular step is identified.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim rests on several domain assumptions about contract structure, regex routing, and embedding model adequacy. None are tested with ablations or comparative experiments. No new physical or formal entities are introduced.

free parameters (1)

embedding dimension = 1536
The authors selected 1536-dimensional text-davinci-002 embeddings by hand; no comparison against other dimensions is reported, so the claimed accuracy depends on this unexamined choice.

assumptions (4)

domain assumption Contracts have a standardized section structure, so chunking by clauses yields semantically complete units.
Section 3.1 states 'sections with the same numbering or in the same vicinity describe the same contractual aspect... we can assume that the best chunking strategy is to separate the chunks by section.' If this fails, RAG retrieval quality degrades.
domain assumption Router Agent's hand-written regular expressions can classify queries into RAG, SQL, or graph paths with sufficient accuracy.
Section 3.3 says the router uses regular expressions to identify keywords and patterns; Section 5 concedes that integrating machine learning models could increase precision and flexibility, implying current routing is fragile.
domain assumption The embedding model text-davinci-002 with 1536 dimensions captures semantic alignment between queries and contract text.
Section 3.1 states the authors selected this model and dimension after considering alternatives, without comparative experiments; the central accuracy claim depends on this unvalidated choice.
domain assumption The SQLite database schema and text-to-SQL translation produce exact answers for indirect questions.
Section 4 reports indirect questions are answered via the SQL agent, and some indirect questions returned incomplete answers, so this assumption is only partially met.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Dynamic Multi-Agent Orchestration and Retrieval for Multi-Source Question-Answer Systems using Large Language Models." pith.science (2026). https://pith.science/paper/Z4GYQQY3

@misc{pith2026241217964,
  author       = {Pith},
  title        = {Pith review of: Dynamic Multi-Agent Orchestration and Retrieval for Multi-Source Question-Answer Systems using Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z4GYQQY3}},
  note         = {Machine review of arXiv:2412.17964}
}

read the original abstract

We propose a methodology that combines several advanced techniques in Large Language Model (LLM) retrieval to support the development of robust, multi-source question-answer systems. This methodology is designed to integrate information from diverse data sources, including unstructured documents (PDFs) and structured databases, through a coordinated multi-agent orchestration and dynamic retrieval approach. Our methodology leverages specialized agents-such as SQL agents, Retrieval-Augmented Generation (RAG) agents, and router agents - that dynamically select the most appropriate retrieval strategy based on the nature of each query. To further improve accuracy and contextual relevance, we employ dynamic prompt engineering, which adapts in real time to query-specific contexts. The methodology's effectiveness is demonstrated within the domain of Contract Management, where complex queries often require seamless interaction between unstructured and structured data. Our results indicate that this approach enhances response accuracy and relevance, offering a versatile and scalable framework for developing question-answer systems that can operate across various domains and data sources.

Figures

Figures reproduced from arXiv: 2412.17964 by the authors.

**Figure 1.** Retrieval-Augmented Generation. RAG enables the retrieval of relevant information from large volumes of unstructured text, while Text-to-SQL facilitates precise access to structured data within relational databases. Dynamic Prompt Engineering customizes the query context, ensuring that responses are tailored to user intent, and Agent-based orchestration coordinates these techniques, directing queries to the approp… view at source ↗

**Figure 2.** Chunking based on Contract’s clauses Due to the similarity between the query and these irrelevant sections, the Retrieval-Augmented Generation (RAG) system may retrieve a chunk from one of these irrelevant clauses that does not actually contain the required name. For example, instead of retrieving the clause that explicitly names the contract manager, the system might retrieve a clause that discusses the general … view at source ↗

**Figure 3.** Chunk’s metadata Embeddings models Embedding models are a cornerstone of modern NLP tasks and plays in importante role in our methodology. These models transform words, sentences, or even entire documents into high-dimensional vectors, or embeddings, and the key advantage of embeddings is that they enable more nuanced and semantically aware operations on text data, such as similarity comparisons and clustering. B… view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Agents Architecture. The Router Agent uses regular expressions to identify keywords, patterns, or structures within the query. If the query is specific to a clause within a contract, the Router Agent recognizes this pattern and assigns the query to the RAG Agent. The …

**Figure 5.** Figure 5: Application architecture. The Backend Agents act as the decision-making layer of the system, handling queries based on their type and content. These agents include the Router Agent, which determines whether to route the query to the RAG Agent (for unstructured text ret…

**Figure 6.** Figure 6: Contracts Q&A Streamlit application contracts, including both PDFs and associated metadata, were processed to assess the system’s ability to retrieve relevant information from both unstructured documents and structured data. To evaluate the system’s effectiveness in an…

**Figure 7.** Figure 7: Plotly Agent 5 Conclusions and Future Work In this work, we presented a comprehensive multi-source question-answer system that integrates unstructured text from contract documents with structured data from relational databases. By employing a combination of Retrieval-A…

**Figure 8.** Figure 8: Contract Summarization The 8 demonstrates the ability of Contrato360 in retrieving and summarizing contract information related to Oracle through a question-and-answer interface. Our implementation, which includes the use of ChromaDB as the vectorstore for storing doc…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 7 canonical work pages

[2]

Feng, Z., Feng, X., Zhao, D., Yang, M., and Qin, B. (2024). Retrieval-generation synergy augmented large language models. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 11661–11665. IEEE. Gao et al., 2023a. Gao, D., Wang, H., Li, Y., Sun, X., Qian, Y., Ding, B., and Zhou, J. (2023a). Text-to-s...

arXiv 2024
[4]

Jeong, C. (2023). A study on the implementation of generative ai services using an enterprise data-based llm application architecture. arXiv preprint arXiv:2309.01105 . Jin et al.,

arXiv 2023
[5]

Jin, H., Huang, L., Cai, H., Yan, J., Li, B., and Chen, H. (2024). From llms to llm- based agents for software engineering: A survey of current, challenges and future. arXiv preprint arXiv:2408.02479. Lewis et al.,

arXiv 2024
[7]

Liu, A., Hu, X., Wen, L., and Yu, P. S. (2023). A comprehensive evaluation of chatgpt’s zero-shot text-to-sql capability. arXiv preprint arXiv:2303.13547 . Mialon et al.,

arXiv 2023
[8]

Mialon, G., Dess ` ı, R., Lomeli, M., Nalmpantis, C., Pasunuru, R., Raileanu, R., Rozi` ere, B., Schick, T., Dwivedi-Yu, J., Celikyilmaz, A., et al. (2023). Augmented language models: a survey. arXiv preprint arXiv:2302.07842 . OpenAI, 2023a. OpenAI (2023a). Chatgpt fine-tune description. https://help.openai.com/en/ articles/6783457-what-is-chatgpt . Acce...

arXiv 2023
[9]

Pinheiro, J., Victorio, W., Nascimento, E., Seabra, A., Izquierdo, Y., Garcıa, G., Coelho, G., Lemos, M., Leme, L. A. P. P., Furtado, A., et al. (2023). On the construction of database interfaces based on large language models. In Proceedings of the 19th International Conference on Web Information Systems and Technologies - Volume 1: WEBIST , pages 373–38...

work page 2023
[10]

Seabra, A., Nepomuceno, J., Lago, L., Ruberg, N., and Lifschitz, S. (2024). Con- trato360: uma aplica¸ c˜ ao de perguntas e respostas usando modelos de linguagem, documentos e bancos de dados. In Anais do XXXIX Simp´ osio Brasileiro de Bancos de Dados. Singh et al.,

work page 2024
[11]

Singh, A., Ehtesham, A., Kumar, S., and Khoei, T. T. (2024). Enhancing ai systems with agentic workflows patterns in large language model. In 2024 IEEE World AI IoT Congress (AIIoT) , pages 527–532. IEEE. Wang et al.,

work page 2024

Show all 13 references

[12]

Wang, M., Wang, M., Xu, X., Yang, L., Cai, D., and Yin, M. (2023). Unleashing chatgpt’s power: A case study on optimizing information retrieval in flipped classrooms via prompt engineering. IEEE Transactions on Learning Technologies. White et al.,

2023
[13]

White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., Elnashar, A., Spencer- Smith, J., and Schmidt, D. C. (2023). A prompt pattern catalog to enhance prompt engineering with chatgpt. arXiv preprint arXiv:2302.11382 . Authors Antony Seabra is an IT executive at BND...

2023 arXiv
[2020]

Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., K¨ uttler, H., Lewis, M., Yih, W.-t., Rockt¨ aschel, T., et al. (2020). Retrieval-augmented generation for knowledge- intensive nlp tasks. Advances in Neural Information Processing Systems , 33:9459–9474....

2020
[2023]

Giray, L. (2023). Prompt engineering with chatgpt: a guide for academic writers. Annals of biomedical engineering, 51(12):2629–2633. Jeong,

2023
[2024]

Chen, J., Lin, H., Han, X., and Sun, L. (2024). Benchmarking large language models in retrieval-augmented generation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 17754–17762. Feng et al.,

2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [2]

Feng, Z., Feng, X., Zhao, D., Yang, M., and Qin, B. (2024). Retrieval-generation synergy augmented large language models. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 11661–11665. IEEE. Gao et al., 2023a. Gao, D., Wang, H., Li, Y., Sun, X., Qian, Y., Ding, B., and Zhou, J. (2023a). Text-to-s...

arXiv 2024

[2] [4]

Jeong, C. (2023). A study on the implementation of generative ai services using an enterprise data-based llm application architecture. arXiv preprint arXiv:2309.01105 . Jin et al.,

arXiv 2023

[3] [5]

Jin, H., Huang, L., Cai, H., Yan, J., Li, B., and Chen, H. (2024). From llms to llm- based agents for software engineering: A survey of current, challenges and future. arXiv preprint arXiv:2408.02479. Lewis et al.,

arXiv 2024

[4] [7]

Liu, A., Hu, X., Wen, L., and Yu, P. S. (2023). A comprehensive evaluation of chatgpt’s zero-shot text-to-sql capability. arXiv preprint arXiv:2303.13547 . Mialon et al.,

arXiv 2023

[5] [8]

Mialon, G., Dess ` ı, R., Lomeli, M., Nalmpantis, C., Pasunuru, R., Raileanu, R., Rozi` ere, B., Schick, T., Dwivedi-Yu, J., Celikyilmaz, A., et al. (2023). Augmented language models: a survey. arXiv preprint arXiv:2302.07842 . OpenAI, 2023a. OpenAI (2023a). Chatgpt fine-tune description. https://help.openai.com/en/ articles/6783457-what-is-chatgpt . Acce...

arXiv 2023

[6] [9]

Pinheiro, J., Victorio, W., Nascimento, E., Seabra, A., Izquierdo, Y., Garcıa, G., Coelho, G., Lemos, M., Leme, L. A. P. P., Furtado, A., et al. (2023). On the construction of database interfaces based on large language models. In Proceedings of the 19th International Conference on Web Information Systems and Technologies - Volume 1: WEBIST , pages 373–38...

work page 2023

[7] [10]

Seabra, A., Nepomuceno, J., Lago, L., Ruberg, N., and Lifschitz, S. (2024). Con- trato360: uma aplica¸ c˜ ao de perguntas e respostas usando modelos de linguagem, documentos e bancos de dados. In Anais do XXXIX Simp´ osio Brasileiro de Bancos de Dados. Singh et al.,

work page 2024

[8] [11]

Singh, A., Ehtesham, A., Kumar, S., and Khoei, T. T. (2024). Enhancing ai systems with agentic workflows patterns in large language model. In 2024 IEEE World AI IoT Congress (AIIoT) , pages 527–532. IEEE. Wang et al.,

work page 2024