{"id":"af5f6c43-9abc-4bad-a68a-00357635863c","arxiv_id":"2605.27331","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":4.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Maat is a ReAct agent that orchestrates tools and RAG for competition law research, outperforming baselines on case-specific tasks while providing official citations.","lead":"Maat is a ReAct agent for competition law research that uses RAG to ground answers in official sources, provides citations, and falls back to web search. A smart generalist might read it to see how domain-specific agents can reduce hallucinations in professional legal tasks.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"RAG/web-search reliability for official citations is the load-bearing assumption for the outperformance claim","rationale":"The reader's weakest assumption directly identifies the same point that must hold for the central empirical claim to be credible. Because the full manuscript still supplies no error analysis or citation-verification protocol, the concern remains load-bearing and the UNVERDICTED / LOW verdict is appropriate.","tokens_in":1663,"tokens_out":310,"duration_ms":19354,"concrete_test":"Sample 30 case-specific queries from the reported test set; for each Maat output, independently verify every citation against the official source (EUR-Lex, national competition authority databases); compute citation error rate (wrong case, wrong paragraph, fabricated decision). If error rate exceeds 8 % or if >15 % of queries have at least one uncorrectable citation error, re-score the task accuracy excluding those queries.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline result (Maat significantly outperforms baselines on case-specific tasks) rests on the system delivering accurate, official-sourced answers with rich citations. The description states that RAG grounds findings in official sources and falls back to web search, yet the evaluation section provides no quantitative measurement of citation error rate, coverage gaps, or hallucinated cases. If retrieval misses key precedents or returns inaccurate citations on a non-negligible fraction of queries, the measured performance delta could be driven by evaluator leniency or task selection rather than genuine reliability gains.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces Maat, a ReAct agent for competition-law research that orchestrates specialized tools, uses RAG to ground answers in official sources with rich citations, falls back to web search for coverage gaps, and seeks user clarification on ambiguous queries. It claims that Maat significantly outperforms general assistants (Claude, ChatGPT) and legal assistants (SaulLM-7B, LegalGPT) on case-specific tasks while performing within the range of the top baseline on theoretical questions; a dataset is released on GitHub.","tokens_in":1747,"tokens_out":565,"duration_ms":22309,"significance":"If the performance claims were supported by rigorous, reproducible evaluation, the work would demonstrate a practical advance in domain-specialized agentic systems for legal research by showing how tool orchestration and source grounding can reduce hallucinations relative to general LLMs.","major_comments":[{"comment":"Evaluation section (and abstract): the headline claim that Maat 'significantly outperforms all baseline assistants on case-specific tasks' is presented without any reported metrics, baseline descriptions, statistical tests, error analysis, or task counts, so the data cannot be checked against the claim.","section":"Evaluation / Abstract"},{"comment":"RAG and web-search description: the system is said to 'ground cases and findings in official sources using RAG' and to 'provide rich in-line citations,' yet no quantitative measurement of citation error rate, coverage gaps, or hallucinated precedents is supplied; this is load-bearing for the reliability and outperformance assertions.","section":"System description / Evaluation"},{"comment":"Dataset and reproducibility: while the dataset is stated to be available on GitHub, the paper supplies no details on how the case-specific and theoretical tasks were constructed, how baselines were prompted or evaluated, or any inter-annotator agreement for the expert-designed tasks.","section":"Dataset / Evaluation"}],"minor_comments":[{"comment":"The abstract and introduction repeatedly use 'competition protection' and 'competition law' interchangeably without clarifying whether the scope is limited to EU, US, or multi-jurisdictional sources.","section":"Abstract / Introduction"},{"comment":"No explicit list or description of the 'tools corresponding to different tasks of the research process' is provided, making the ReAct orchestration hard to replicate from the text alone.","section":"System architecture"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be an early or incomplete draft; the absence of any quantitative results in the provided sections suggests the evaluation may have been omitted or is only in supplementary material not referenced here."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the careful and constructive review. The comments highlight important gaps in the presentation of evaluation results, system reliability metrics, and methodological details. We address each point below and will revise the manuscript to provide the requested rigor and transparency.","responses":[{"response":"We agree that the current version of the manuscript does not supply the quantitative details needed to substantiate the performance claims. In the revision we will expand the evaluation section (and update the abstract) to report concrete metrics, full baseline prompting and evaluation protocols, task counts, statistical significance tests, and error analysis.","revision_made":"yes","referee_comment":"[Evaluation / Abstract] Evaluation section (and abstract): the headline claim that Maat 'significantly outperforms all baseline assistants on case-specific tasks' is presented without any reported metrics, baseline descriptions, statistical tests, error analysis, or task counts, so the data cannot be checked against the claim."},{"response":"The observation is correct: the manuscript currently lacks quantitative assessment of citation accuracy, coverage, or hallucination rates. We will add an evaluation subsection that measures citation error rates (via expert review or automated verification against official sources), quantifies coverage gaps, and reports any detected hallucinated precedents.","revision_made":"yes","referee_comment":"[System description / Evaluation] RAG and web-search description: the system is said to 'ground cases and findings in official sources using RAG' and to 'provide rich in-line citations,' yet no quantitative measurement of citation error rate, coverage gaps, or hallucinated precedents is supplied; this is load-bearing for the reliability and outperformance assertions."},{"response":"We will insert a new subsection that fully describes task construction (including expert involvement), baseline prompting templates and evaluation procedures, and any inter-annotator or expert-validation steps used. The GitHub repository will be updated with corresponding documentation and scripts.","revision_made":"yes","referee_comment":"[Dataset / Evaluation] Dataset and reproducibility: while the dataset is stated to be available on GitHub, the paper supplies no details on how the case-specific and theoretical tasks were constructed, how baselines were prompted or evaluated, or any inter-annotator agreement for the expert-designed tasks."}],"tokens_in":1344,"tokens_out":493,"duration_ms":35552,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"Maat builds a ReAct agent that breaks legal research into tool calls, grounds answers in official competition-law sources via RAG, adds inline citations, and asks for clarification on vague queries. The authors iterated with domain experts and released the evaluation dataset on GitHub.\n\nWhat stands out is the narrow focus on competition and merger cases plus the practical handling of coverage gaps through web search. That combination is new for this regulated slice of law even though the underlying ReAct and RAG pieces are standard.\n\nThe main gap is the results. The abstract states clear outperformance on case-specific tasks and parity on theory questions, yet it gives no accuracy numbers, no baseline details, no statistical tests, and no measurement of how often citations are wrong or missing. Without those, the claimed advantage cannot be checked and could trace to task choice or lenient judging rather than retrieval quality. The stress-test note correctly flags RAG reliability as the load-bearing assumption here.\n\nThe work is aimed at legal-tech developers who need a starting point for specialized assistants in regulated domains. A reader already building RAG systems for law could pull the system description and the public dataset for their own experiments.\n\nI would send the paper to peer review if the authors add the missing quantitative evaluation and error analysis; the idea is concrete enough and the domain narrow enough that referees could give useful feedback on the implementation.","headline":"Maat is a ReAct agent for competition law that uses RAG plus web fallback and claims better case-task results, but the evaluation supplies no metrics or citation-error checks so the gains stay unverified.","tokens_in":2228,"tokens_out":366,"would_cite":false,"duration_ms":25003,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Maat is a ReAct agent that grounds competition law research in official sources via RAG and web fallback, outperforming general assistants on case-specific tasks.","keywords":["competition law","legal research assistant","ReAct agent","retrieval-augmented generation","case precedent analysis","official source grounding","agentic AI for regulation"],"falsifier":"A blind test set of competition-law queries where Maat produces a higher rate of incorrect case citations or fabricated precedents than the strongest baseline.","tokens_in":2555,"feed_emoji":"⚖️","tokens_out":628,"duration_ms":20626,"temperature":0.7,"pith_summary":"The paper introduces Maat as a specialized agent built with competition law experts to handle the volume of cases and decisions required for precedent analysis and merger assessment. It claims that orchestrating tools through ReAct, retrieving from official databases with RAG, supplying inline citations, and falling back to web search when needed produces more reliable outputs than general models like Claude or ChatGPT or other legal assistants. The evaluation shows clear gains on tasks that require specific case handling while staying competitive on broader theoretical questions. A dataset is released to support further work.","feed_headline":"Specialized agent outperforms general models on competition law cases","feed_subtitle":"Maat uses ReAct orchestration and RAG over official sources to improve accuracy on case-specific research tasks.","key_machinery":"ReAct agent that orchestrates task-specific tools, combined with RAG over official competition-law sources for citations and a web-search fallback for coverage gaps.","core_discovery":"Maat significantly outperforms all baseline assistants on case-specific tasks and performs within range of the top baseline on theoretical question tasks by using a ReAct loop to call specialized tools, retrieving from an official-source RAG index for grounding and citations, and invoking web search only when database coverage is insufficient.","pith_inferences":["If the reliability claims hold, similar agent designs could shorten the review cycle for merger filings that currently require extensive manual case retrieval.","Extending the RAG index to include more recent decisions would be a direct next step to test whether performance gains persist as the case corpus grows.","The approach suggests that tool orchestration plus source grounding may be more important than model size alone for domain-specific legal accuracy."],"forward_implications":["Competition-law practitioners could reduce time spent manually cross-checking precedents while maintaining traceability to official documents.","The same agent structure could be adapted to other regulatory domains that rely on large bodies of case decisions.","Releasing the evaluation dataset allows direct comparison of future agents on the same case-specific and theoretical benchmarks.","Prompting users for clarification on ambiguous queries reduces downstream errors in precedent identification."],"fun_headline_variants":["Maat outperforms baselines on case-specific competition tasks","ReAct agent Maat grounds results in official legal sources","Maat uses RAG for reliable competition law case citations","Agentic Maat tops models on specific legal research tasks"],"cache_read_input_tokens":64,"weakest_assumption_plain":"The RAG index and web-search fallback will retrieve and cite only accurate official sources without coverage gaps or errors that change the measured performance advantage.","fun_headline_variants_meta":{"raw":{"variants":["Maat outperforms baselines on case-specific competition tasks","ReAct agent Maat grounds results in official legal sources","Maat uses RAG for reliable competition law case citations","Agentic Maat tops models on specific legal research tasks"]},"model":"grok-4.3","cost_usd":0.004213,"raw_usage":{"total_tokens":2010,"prompt_tokens":596,"num_sources_used":0,"completion_tokens":63,"cost_in_usd_ticks":42128000,"prompt_tokens_details":{"text_tokens":596,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1351,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":596,"tokens_out":63,"duration_ms":15293,"temperature":1.0,"reasoning_tokens":1351,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-29T16:33:42.491889+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A blind test set of competition-law queries where Maat produces a higher rate of incorrect case citations or fabricated precedents than the strongest baseline.","supporting_citations":[],"review_version":1}