{"id":"9a51316b-b0c2-4a15-b854-061df7820ee0","arxiv_id":"2605.30862","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Sophrosyne augments fine-grained data APIs with directives to curb over-exploration by Text2SQL agents, reducing it 4.6x and improving accuracy up to 4 points.","lead":"Text2SQL LLM agents using fine-grained data system APIs over-explore schemas and produce inaccurate queries. Sophrosyne adds guiding directives to API responses to moderate exploration, cutting over-exploration 4.6x and raising accuracy by roughly 4 percentage points.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"No significant objection identified","rationale":"The reader's provisional UNVERDICTED assessment matches the information available; the abstract alone does not contain enough technical detail to surface a load-bearing flaw in the argument.","tokens_in":1709,"tokens_out":206,"duration_ms":12815,"concrete_test":"Retrieve the full manuscript and examine the experimental section (likely §4 or §5) for the exact benchmark suite, number of queries, baseline agent configurations, and statistical tests; recompute the 4.6x and 12.4% figures from the raw per-query data if available.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract states a clear hypothesis (fine-grained APIs cause over-exploration leading to inaccurate Text2SQL) and reports initial quantitative gains from directives, but supplies no methods, benchmarks, or controls. Without the full text, no internal inconsistency, hidden assumption in a derivation, or unsupported empirical claim can be isolated.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper argues that most data systems expose fine-grained APIs, which cause LLM-based Text2SQL agents to over-explore by incorporating irrelevant schema elements and generating inaccurate queries. It proposes Sophrosyne, a data system environment that augments API responses with directives to moderate the agent's exploration process, and reports initial results showing a 4.6x reduction in over-exploration and up to 12.4% (approximately 4 percentage points) accuracy improvement.","tokens_in":1735,"tokens_out":434,"duration_ms":18446,"significance":"If the empirical results hold under rigorous evaluation, the work could be significant for the design of API surfaces in relational data systems intended for agentic use, by formalizing a cost-accuracy tradeoff between coarse- and fine-grained APIs and demonstrating a lightweight moderation mechanism via directives. The approach of embedding guidance directly in API responses is a practical contribution that could be adopted in production systems.","major_comments":[{"comment":"Abstract: The central empirical claims (4.6x reduction in over-exploration and up to 12.4% accuracy boost) are presented as direct outcomes of the proposed directives, yet the abstract supplies no experimental setup, benchmark datasets, agent implementation details, baseline systems, definition or measurement protocol for 'over-exploration,' or statistical analysis. This is load-bearing because the soundness of the hypothesis and the value of Sophrosyne rest entirely on these unverified quantitative results.","section":"Abstract"},{"comment":"Abstract: The posited fundamental tradeoff between coarse-grained and fine-grained APIs is asserted without any formal definitions, concrete examples drawn from existing data systems, or analysis showing how fine-grained APIs specifically induce over-exploration in Text2SQL agents. This weakens the motivation for the proposed moderation technique.","section":"Abstract"}],"minor_comments":[{"comment":"The abstract contains a subject-verb agreement error ('they over-explore ... and produce inaccurate results').","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback on the abstract and the recommendation for major revision. We agree that the abstract can be strengthened to better contextualize the empirical claims and motivation. We respond to each major comment below and will incorporate revisions in the next manuscript version.","responses":[{"response":"We acknowledge that the abstract's brevity omits key experimental context, which could make the quantitative claims harder to evaluate at a glance. The full manuscript details the evaluation in Sections 4 and 5: benchmarks include Spider and BIRD; over-exploration is defined and measured as the rate of irrelevant schema elements incorporated (via schema element precision/recall); the agent uses a ReAct-style loop with tool calling; baselines include vanilla LLM prompting and unmoderated fine-grained API access; results include statistical analysis via multiple runs and significance testing. To address the concern directly, we will revise the abstract to concisely incorporate a high-level description of the setup, the over-exploration metric, and the evaluation protocol while remaining within length limits.","revision_made":"yes","referee_comment":"[Abstract] Abstract: The central empirical claims (4.6x reduction in over-exploration and up to 12.4% accuracy boost) are presented as direct outcomes of the proposed directives, yet the abstract supplies no experimental setup, benchmark datasets, agent implementation details, baseline systems, definition or measurement protocol for 'over-exploration,' or statistical analysis. This is load-bearing because the soundness of the hypothesis and the value of Sophrosyne rest entirely on these unverified quantitative results."},{"response":"Section 2 of the manuscript categorizes real-world APIs from systems including PostgreSQL (fine-grained introspection endpoints), MySQL, BigQuery (coarser query interfaces), and Snowflake, with concrete examples of how fine-grained surfaces enable step-by-step schema exploration that leads agents to include extraneous tables/columns. Section 3 provides formal definitions of coarse- vs. fine-grained APIs and formalizes over-exploration as unnecessary expansion of the explored schema subgraph. We will revise the abstract to include a brief clause referencing this tradeoff analysis and its grounding in existing systems to better motivate the directives approach.","revision_made":"yes","referee_comment":"[Abstract] Abstract: The posited fundamental tradeoff between coarse-grained and fine-grained APIs is asserted without any formal definitions, concrete examples drawn from existing data systems, or analysis showing how fine-grained APIs specifically induce over-exploration in Text2SQL agents. This weakens the motivation for the proposed moderation technique."}],"tokens_in":1334,"tokens_out":541,"duration_ms":23874,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The punchline is that this work identifies a real tension in how data systems expose APIs to LLM agents and offers a lightweight moderation fix via response directives. It frames the choice between coarse and fine-grained APIs as a cost-accuracy tradeoff and argues that fine-grained ones, which most systems use for scoping, lead agents to pull in irrelevant schema bits and produce worse SQL.\n\nWhat is new is the specific tactic of embedding exploration directives directly in API replies rather than altering the API surface or adding separate guardrails. The paper does a clear job spelling out why over-exploration happens and why curbing it matters for practical Text2SQL use.\n\nThe soft spot is the evidence. The abstract reports the 4.6x and 12.4% gains as initial results but supplies no experimental setup, agent details, datasets, baselines, or measurement definitions. Without those, it is impossible to judge whether the directives are the cause or whether the numbers would hold under different conditions. The central claim therefore rests on unverified outcomes.\n\nThis is aimed at people working on LLM-database interfaces or agentic query systems. A reader in that niche could pick up the moderation idea and test it themselves. The work shows honest engagement with the practical constraint but stays at the level of a targeted proposal.\n\nI would send it to peer review if the full paper includes proper controls and comparisons; the idea is narrow enough that referees could evaluate it quickly. Otherwise it reads more like a workshop note than a finished result.","headline":"The paper's main claim is that adding directives to fine-grained DB API responses cuts agent over-exploration in Text2SQL by 4.6x and lifts accuracy by ~4 points, but the abstract gives no methods or baselines to check those numbers.","tokens_in":2223,"tokens_out":401,"would_cite":false,"duration_ms":20636,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Fine-grained APIs in data systems cause Text2SQL agents to over-explore schemas and generate inaccurate queries.","keywords":["Text2SQL","LLM agents","database APIs","over-exploration","query generation","data systems","directives","Sophrosyne"],"falsifier":"Run the same Text2SQL agent on a fixed benchmark set of natural language questions both with and without Sophrosyne directives, then count the number of irrelevant schema elements referenced in the generated queries.","tokens_in":2602,"feed_emoji":"","tokens_out":602,"duration_ms":20443,"temperature":0.7,"pith_summary":"Text2SQL agents use LLM-powered tool calls to explore data system APIs and gather schema details before writing SQL from natural language. Most systems expose fine-grained APIs for security, but this leads agents to pull in irrelevant schema elements and produce wrong queries. The paper argues that curbing over-exploration is essential for making these APIs usable by agents. It proposes Sophrosyne, an environment that adds directives to API responses to steer the exploration. Early tests show the directives cut over-exploration by 4.6 times and raise accuracy by as much as 12.4 percent.","feed_headline":"Directives cut database agent over-exploration by 4.6 times","feed_subtitle":"Guiding LLM exploration of fine-grained APIs improves Text2SQL accuracy by up to 12.4 percent.","key_machinery":"Sophrosyne, a data system environment that augments API responses with directives to guide the agent's exploration process.","core_discovery":"Text2SQL agents explore relational data systems through fine-grained APIs before formulating queries, yet this causes them to incorporate irrelevant schema elements and yield inaccurate SQL. Sophrosyne augments API responses with directives that guide the agent's exploration process, reducing over-exploration by 4.6x and improving accuracy by up to 12.4 percent.","pith_inferences":["The same directive approach could moderate agent behavior when interacting with APIs in non-relational or non-database systems.","API designers may need to consider agent exploration patterns as a first-class requirement alongside human users.","Standardized directive formats across data systems could reduce the need for per-system solutions like Sophrosyne."],"forward_implications":["Agents can safely use fine-grained APIs without the accuracy penalty from over-exploration.","Data systems maintain their preferred fine-grained access model while supporting agentic workloads.","Fewer tool calls occur during exploration, lowering the cost of query formulation.","Accuracy on standard Text2SQL tasks rises without changes to the base API surface."],"fun_headline_variants":["Directives cut Text2SQL over-exploration 4.6x","Sophrosyne moderates agent exploration of data APIs","Directives improve Text2SQL by curbing over-exploration","Over-exploration reduced 4.6x with moderated APIs"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Agents will incorporate the added directives into their exploration behavior instead of continuing to over-explore.","fun_headline_variants_meta":{"raw":{"variants":["Directives cut Text2SQL over-exploration 4.6x","Sophrosyne moderates agent exploration of data APIs","Directives improve Text2SQL by curbing over-exploration","Over-exploration reduced 4.6x with moderated APIs"]},"model":"grok-4.3","cost_usd":0.013639,"raw_usage":{"total_tokens":5867,"prompt_tokens":608,"num_sources_used":0,"completion_tokens":69,"cost_in_usd_ticks":136387000,"prompt_tokens_details":{"text_tokens":608,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":5190,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":608,"tokens_out":69,"duration_ms":43407,"temperature":1.0,"reasoning_tokens":5190,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-28T20:44:44.715341+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Run the same Text2SQL agent on a fixed benchmark set of natural language questions both with and without Sophrosyne directives, then count the number of irrelevant schema elements referenced in the generated queries.","supporting_citations":[],"review_version":1}