{"id":"a1f82bb6-6e07-4233-9901-e0c03c1c5b5a","arxiv_id":"2412.17964","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"A multi-agent QA system routes contract queries to RAG or SQL agents using regex rules and dynamic prompts, but supports its accuracy claim only with qualitative user feedback.","lead":"This paper describes a question-answer system for contract management that combines several known LLM techniques: retrieval from PDFs, text-to-SQL over a database, and agent-based routing to pick between them. It claims this combination improves accuracy and relevance, but the evidence is qualitative feedback from two specialists rather than a measured benchmark.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The empirical claim of enhanced accuracy/relevance is unsupported: Section 4 reports only qualitative feedback from two specialists with no baseline or metrics, so the central 'enhances' claim is not demonstrated.","rationale":"The reader correctly identifies regex-based routing as a weak link, and Section 5 explicitly acknowledges it. However, the more load-bearing concern is that the central claim is a superiority claim without a comparator. Even a perfect router would not validate 'enhances response accuracy and relevance' because the evaluation is purely qualitative, with two specialists and no baseline. The paper's own Section 4 describes only 'feedback' and 'results demonstrated' without metrics; Section 5 reinforces the limitation. Thus the evidence is insufficient for the stated claim. This does not change the reader's REJECT verdict, but it shifts the emphasis from a technical implementation risk to an evidential gap. The proposed benchmark directly tests both the routing accuracy and the comparative enhancement claim, making the concern empirically settleable.","tokens_in":13774,"tokens_out":2558,"duration_ms":25320,"concrete_test":"Construct a benchmark of 100 queries (50 direct, 50 indirect) over the 75 contracts. Run the proposed multi-agent system against two baselines: (a) a single RAG pipeline over PDFs+metadata only, and (b) a single LLM with access to both vectorstore and SQL via general tool-calling (no regex router). Have independent annotators score factual correctness and relevance (e.g., 1-5) with blinded responses, and compute inter-annotator agreement. Also log the router's predicted agent for each query to compute routing accuracy. If the multi-agent system's scores are not significantly higher than both baselines (e.g., paired bootstrap or Wilcoxon), the central enhancement claim fails; if routing accuracy is low, the regex router is the bottleneck.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in the abstract is that the methodology 'enhances response accuracy and relevance.' To support this comparative claim, the evaluation must measure accuracy/relevance against a baseline. Section 4 states that two IT contract specialists from BNDES validated the system on 75 contracts, with 'direct' and 'indirect' question categories, but provides no quantitative metrics, no baseline system, no inter-rater agreement, and no statistical test. The only reported shortcomings are qualitative (incomplete answers for 'Waiver of Bidding'), and Section 5 concedes the Router Agent 'relies on predefined regular expressions.' Even if regex routing worked perfectly, the paper still lacks evidence that the orchestrated system outperforms a single retrieval pipeline; the 'enhances' wording is therefore a claim without a comparator. The load-bearing assumption is that subjective specialist feedback on one deployment is sufficient to establish superiority, which is not a correctness assumption but an evidential one.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multi-agent orchestration and retrieval methodology for multi-source question-answer systems, combining retrieval-augmented generation (RAG) for unstructured PDF documents, text-to-SQL for structured databases, dynamic prompt engineering, and a router agent that uses predefined regular expressions to select the retrieval path. The methodology is instantiated in Contrato360, a contract-management QA system, and is evaluated through qualitative feedback from two IT contract specialists from BNDES on a set of 75 contracts with direct and indirect questions. The authors claim that the approach enhances response accuracy and relevance compared with traditional single-pipeline systems.","tokens_in":13916,"tokens_out":3782,"duration_ms":32353,"significance":"If the evaluation were rigorous, the paper would provide a useful case study of integrating RAG, text-to-SQL, and agent-based routing in a real-world contract-management domain. The system design, especially the metadata-filtered section-based chunking, is plausible and may be of practical interest to practitioners. However, as presented, the contribution is primarily an engineering integration of known components: there are no new algorithms, no machine-checked proofs, no reproducible benchmark, and no quantitative evaluation. The evidence for the headline claim of enhanced accuracy and relevance is anecdotal and does not yet meet the standard of a journal paper.","major_comments":[{"comment":"The abstract claims that the proposed approach 'enhances response accuracy and relevance,' but Section 4 reports only qualitative feedback from two IT contract specialists, with no baseline system, no quantitative accuracy or relevance metrics, no inter-rater agreement, and no statistical test. Because the central claim is comparative ('enhances'), a comparator is required. Please add a controlled comparison against at least one baseline pipeline (e.g., RAG-only or SQL-only), define and measure accuracy/relevance metrics (e.g., exact match, F1, or a rubric-based score), and report evaluator agreement or a structured scoring procedure. Without this, the claim is unsupported by the presented evidence.","section":"Section 4 (Evaluation)"},{"comment":"The Router Agent is a load-bearing component of the architecture: it uses predefined regular expressions to decide between the RAG and SQL paths, and Section 5 concedes that the router 'relies on predefined regular expressions to route queries' and proposes machine learning as future work. If a query does not match the patterns, the wrong agent is selected and the system's accuracy degrades. The paper provides no data on routing accuracy, pattern coverage, or failure cases, and no fallback mechanism is described. Please report the router's precision and recall on the benchmark questions, and describe how out-of-pattern queries are handled in practice.","section":"Section 3.3 and Section 5"},{"comment":"The text states that the authors selected 'text-davinci-002, a model from OpenAI's GPT-3.5 family, along with embeddings with 1536 dimensions.' text-davinci-002 is a completion model, not an embedding model; the configuration as described is not reproducible. Please identify the actual embedding model used (e.g., text-embedding-ada-002), its dimensionality, and the embedding configuration, or clarify whether the system used a separate embedding model for the vectorstore.","section":"Section 3.1 (Embeddings models)"},{"comment":"The evaluation reports that indirect questions such as those involving 'Waiver of Bidding' produced incomplete answers, yet the abstract and conclusion make an unqualified claim of enhanced accuracy and relevance. Please either temper the claim to match the evidence or provide a detailed error analysis and demonstrate that the proposed prompt adjustments resolve the incompleteness. As written, the reported limitations are not reconciled with the positive conclusion.","section":"Section 4 (Results)"}],"minor_comments":[{"comment":"The manuscript contains several typos and incomplete placeholders: 'Figure ??' appears in Sections 3.1 and 3.4, and 'Manhatan' (Section 3.1), 'the the' (Section 3.3), 'lenghty' (Section 3.1), 'prompt'' (Introduction), and 'V ectorstore' (Section 3.1) should be corrected.","section":"General"},{"comment":"The RAG process is described in seven steps, but Figure 1 is not displayed in the manuscript; please either include the figure or remove the reference to it.","section":"Section 3.1"},{"comment":"The paper refers to several figures (e.g., Figures 5–8) that appear to be screenshots without detailed captions; please add explanatory captions and ensure all figures are actually included and sequentially numbered.","section":"Section 4"},{"comment":"The reference list contains inconsistencies (e.g., 'Gao et al., 2023a' and 'Gao et al., 2023b' are interleaved; 'OpenAI, 2023a' cites a help page rather than a citable publication). Please standardize the bibliography.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is closely related to the authors' prior work on Contrato360 (Seabra et al., 2024), and the present manuscript reads largely as an expanded description of that system with an added router agent and dynamic prompts. The methodological novelty relative to that prior work is incremental, and the evaluation is not yet at the level expected for a journal. These concerns are about scope and evidence rather than correctness, but they should be weighed by the editor."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a straightforward system description of Contrato360, a multi-agent QA pipeline that combines RAG, text-to-SQL, dynamic prompts, and a regex-based router for contract management at BNDES. If you treat it as an engineering case study, it's a decent one. If you take the abstract at its word—that the methodology 'enhances response accuracy and relevance'—the paper doesn't earn that claim.\n\nWhat it does well: the writing is clear and honest. The authors explain why naive RAG fails on contract documents (similar-sounding clauses that don't answer the query) and how they use metadata filtering over contract number and clause to narrow retrieval. That's a practical, sensible fix that many practitioners will recognize. The per-agent dynamic prompts and the Plotly graph agent are small but useful additions. The paper also states its limitations openly: the router uses predefined regular expressions, and some indirect questions (e.g., 'Waiver of Bidding') returned incomplete answers.\n\nThe soft spots are real and central. Section 4 is the entire evaluation: two IT contract specialists from the authors' own organization, a set of 75 contracts, and no quantitative metrics, no baseline system, no inter-rater agreement, no statistical test. The phrase 'results indicate that this approach enhances response accuracy and relevance' has no comparator. A single retrieval pipeline, or even a non-agent RAG baseline, would be the obvious thing to test against. The router's regex reliance is another load-bearing assumption; the paper concedes it as future work, but if the regexes fail, the whole routing decision misfires and the accuracy gain collapses. On novelty, the core system is prior work by the same group, and the component techniques are all cited from the literature. The incremental pieces are variations on established agentic patterns.\n\nWho is this for? Practitioners building enterprise QA over mixed PDF/database sources, and readers who want a candid account of a real deployment. It is not a research result that advances the field. I would not send this to a top-tier venue; desk reject is defensible. If the authors add a proper benchmark with public data, baselines, and metrics, it could become a useful application paper.","headline":"A readable engineering write-up of an enterprise contract-QA system, but the central accuracy claim rests on two specialists' qualitative feedback with no baseline; not a research result as it stands.","tokens_in":14468,"tokens_out":2419,"would_cite":false,"duration_ms":25332,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A router agent that dispatches each query to a RAG-based document agent or a text-to-SQL database agent, with prompts adapted per agent, yields more accurate and relevant answers in a multi-source contract-management question-answer system.","keywords":["Information Retrieval","Question Answer","Large Language Models","Documents","Databases","Prompt Engineering","Retrieval Augmented Generation","Text-to-SQL"],"falsifier":"Collect a held-out set of user queries whose wording intentionally avoids the router's known keyword and regex patterns (synonyms, paraphrases, indirect phrasings), run them through the system, and compare the fraction of wrong-agent routes and incorrect answers against a single-pipeline RAG baseline; if routing errors are common and responses degrade accordingly, the central claim fails.","tokens_in":13588,"feed_emoji":"🤖","tokens_out":6445,"duration_ms":54647,"temperature":0.7,"pith_summary":"The paper proposes a question-answering architecture that combines retrieval-augmented generation, text-to-SQL, and dynamically adapted prompts, coordinated by specialized agents. Its central claim is that routing each query to the right agent—RAG for unstructured contract text, SQL for structured database values—produces more accurate and relevant answers than a single generic retrieval pipeline. The system is demonstrated in contract management, where questions often span both PDF documents and relational records. The paper's evidence is qualitative feedback from two domain specialists on a set of 75 contracts, arguing the approach saves time and yields complete answers for direct questions, with some incompleteness on semantically tricky indirect queries.","feed_headline":"Route each contract query to RAG or SQL for sharper answers","feed_subtitle":"Why care: one system answers both document-content and exact-value database questions without retraining.","key_machinery":"The carrying mechanism is the agent-based orchestration pipeline: a Router Agent uses predefined regular expressions to identify query patterns and dispatch each question to the appropriate specialist agent. The RAG Agent retrieves semantically similar chunks from a vector database, filtered by metadata such as source, contract, and clause, to resolve the similar-but-irrelevant ambiguity common in contracts; the SQL Agent translates natural-language questions into exact SQL over a relational database; and a Graph Agent prompts the model to render tabular results as bar graphs. Dynamic prompt engineering tailors the instructions to whichever agent handles the query, adapting the LLM's behavior without retraining.","core_discovery":"The paper's central claim is that a multi-agent orchestration layer, built on a Router Agent that classifies queries by regular expression patterns, can dynamically select between a RAG Agent for unstructured document retrieval via vector embeddings and metadata filtering and a SQL Agent for exact structured querying via text-to-SQL, with dynamic prompts per agent improving response accuracy and contextual relevance. In the contract-management testbed, the authors report that direct questions—those answerable from contract PDFs and their metadata—were consistently answered completely, while indirect questions requiring database values were generally satisfactory, with incomplete answers for terms like 'Waiver of Bidding' traced to routing or semantic retrieval gaps.","pith_inferences":["The qualitative feedback suggests perceived gains, but a quantitative benchmark comparing the multi-agent system against single-pipeline RAG and text-to-SQL baselines on the same query set would directly test and sharpen the claimed improvement.","The regex-based router is the natural failure point; replacing the hand-written patterns with a trained classifier, or with the LLM itself making the routing decision, could make the approach robust to paraphrase and scale to new domains.","The same orchestration pattern should transfer to other mixed-source domains, such as medical records (clinical notes plus structured lab data) or finance (prospectus PDFs plus market databases), where the metadata-chunk filtering idea carries over.","Embedding structured data as text into the vectorstore and querying it via SQL are treated as alternatives; a hybrid that first routes and then fuses results from both paths could handle queries that need both exact values and interpretive context."],"forward_implications":["A system built on this methodology answers both document-content questions and exact-value database questions in one interface, removing the need to search PDFs manually.","Because prompts adapt per agent, the same LLM can be steered for interpretive text synthesis, precise SQL generation, and graphical summaries without retraining.","Metadata filtering on chunks (source, contract, clause) reduces the risk of retrieving semantically similar but irrelevant contract sections.","The modular agent design supports adding new retrieval paths, such as graph or API agents, without rearchitecting the system.","Indirect questions with complex legal semantics, such as 'Waiver of Bidding,' remain a known weak point, and the authors expect prompt or query-structure adjustments to improve them."],"supporting_citations":[{"why":"This previous work introduced Contrato360, the contract-management QA application that the methodology extends and evaluates.","marker":"Seabra et al., 2024"},{"why":"This is the foundational retrieval-augmented generation formulation that the RAG Agent builds upon.","marker":"Lewis et al., 2020"},{"why":"This survey covers RAG chunking and similarity metrics, informing the vectorstore retrieval design in the paper.","marker":"Gao et al., 2023b"},{"why":"This evaluation of LLM zero-shot text-to-SQL capability underlies the SQL Agent's translation approach.","marker":"Liu et al., 2023"},{"why":"This work shows how LLMs construct natural-language database interfaces by mapping entities, the basis for the SQL Agent's query translation.","marker":"Pinheiro et al., 2023"},{"why":"This reference describes agentic workflow patterns that motivate the Router Agent orchestration in the system.","marker":"Singh et al., 2024"}],"fun_headline_variants":["Router agent picks RAG or SQL per query for sharper contract answers","Multi-agent QA: router decides between RAG and SQL dynamically","Two specialized agents, one router: dynamic retrieval for QA","Router agent dynamically chooses RAG or SQL for each query","Dynamic router sends contract questions to best LLM agent"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The Router Agent's predefined regular expressions must correctly classify every user query into the correct retrieval path; a query phrased outside these patterns gets sent to the wrong agent and the claimed accuracy advantage collapses.","fun_headline_variants_meta":{"raw":{"variants":["Router agent picks RAG or SQL per query for sharper contract answers","Multi-agent QA: router decides between RAG and SQL dynamically","Two specialized agents, one router: dynamic retrieval for QA","Router agent dynamically chooses RAG or SQL for each query","Dynamic router sends contract questions to best LLM agent"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000485,"raw_usage":{"total_tokens":2341,"prompt_tokens":843,"completion_tokens":1498,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":459,"completion_tokens_details":{"reasoning_tokens":1415}},"tokens_in":459,"tokens_out":1498,"duration_ms":10893,"temperature":1.0,"reasoning_tokens":1415,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:07:48.941442+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect a held-out set of user queries whose wording intentionally avoids the router's known keyword and regex patterns (synonyms, paraphrases, indirect phrasings), run them through the system, and compare the fraction of wrong-agent routes and incorrect answers against a single-pipeline RAG baseline; if routing errors are common and responses degrade accordingly, the central claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"This previous work introduced Contrato360, the contract-management QA application that the methodology extends and evaluates."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"This is the foundational retrieval-augmented generation formulation that the RAG Agent builds upon."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"This work shows how LLMs construct natural-language database interfaces by mapping entities, the basis for the SQL Agent's query translation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"This reference describes agentic workflow patterns that motivate the Router Agent orchestration in the system."}],"review_version":1}