{"id":"28468de3-26f6-4202-84ff-ca0bf2effbf2","arxiv_id":"2608.12282","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"VAKRA is a benchmark combining executable database-backed APIs, document retrieval, and tool-use policy constraints to evaluate multi-hop reasoning in AI agents, and it finds large performance drops on compositional and policy-constrained tasks.","lead":"The authors introduce VAKRA, a benchmark of more than 7,000 executable APIs and document collections across 62 domains for testing AI agents that reason across tool calls, retrieval, and natural-language policies. It provides a reusable environment and reports that even the strongest model reaches only 70.4% on simple endpoint tasks and substantially lower scores on multi-hop and policy-constrained tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unvalidated LLM judge for the two-stage waterfall is the load-bearing assumption; a human-agreement study on VAKRA's own decisions should settle whether Table 3 scores measure capability.","rationale":"The benchmark's infrastructure is genuinely strong: checksum-verified tool surfaces, deterministic SQLite-backed APIs, one-command Docker reproducibility, re-execution of predicted calls, deterministic Stage 3 policy checks, and a human quality study on generated queries (77–90% annotator agreement). Those pieces support the environment as a reusable measurement instrument. The central claim, however, is about what the model scores measure, and every score in Table 3 passes through the two-stage judge. That judge is used for exactly the kinds of decisions—semantic equivalence of tool-response sets and factual correctness of final answers—where LLM judges are known to be weakest, and the paper provides no VAKRA-specific calibration. I considered the internal inconsistencies noted by the reader (e.g., 'over 8,000' versus 7,087 tools; 644 versus 664 multi-source samples) and the unreported subset for Claude Opus 4.7; these are real documentation problems but they do not threaten the central measurement claim as directly as judge validity. The proposed human-agreement study is feasible and would settle the question: if agreement is high and bias is absent, the headline scores stand; if not, they need to be recomputed. Because this is a concrete, addressable condition rather than a fundamental flaw, the reader's CONDITIONAL verdict is the right one and I would not move it.","tokens_in":29873,"tokens_out":9832,"duration_ms":91886,"concrete_test":"Take a stratified sample of 200 trajectories (50 per setting: Dashboard/SEL/SLOT, Multi-hop, Multi-source; balanced across pass/fail and across models, including GPT-OSS-120B). Have three annotators independently label each trajectory at each gate: (a) Stage 1: do the predicted tool responses recover all ground-truth information? (b) Stage 2: is the final answer correct and grounded in tool responses? Compute exact agreement and Cohen's kappa between GPT-OSS-120B and the human majority per gate and per setting, and run a masked model-identity probe on Stage 2 to test self-preference bias. If per-gate agreement is below ~90% or disagreements concentrate in multi-hop/policy cases, re-score Table 3 using human labels and check whether the reported ranking, the >50% hop degradation, and the 2.4% policy finding survive.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The scoring waterfall in Section 4 is the part of the argument that must be true for the reported numbers to mean what the paper says, and it rests on an unvalidated GPT-OSS-120B judge. Stage 1 uses a programmatic containment check plus an LLM fallback for inconclusive cases; Stage 2 uses a RAGAS-style LLM judge for groundedness and answer correctness. The paper cites prior LLM-as-judge reliability results and sets temperature 0, but it reports no human agreement, no per-stage confusion matrix or error analysis, and no breakdown of how often Stage 1 is resolved programmatically versus by the judge. This matters because the Stage 1 fallback is a semantic-equivalence judgment over sets of tool responses—not the standard chat-quality setting where prior judge reliability was established—and Stage 2 determines the 'Gnd' column in Table 4 and every average in Table 3. GPT-OSS-120B is also itself one of the evaluated models, so potential self-preference bias is unaddressed. If the judge's errors are non-trivial or correlate with reasoning depth or policy category, the headline findings—the 70.4% ceiling, the >50% drop with hops, and the 2.4% unanswerable-policy result—could shift. This is a validation gap, not an integrity problem; the live re-execution and data-quality human study are genuine strengths.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces VAKRA, a benchmark for evaluating multi-hop agentic reasoning across structured APIs and document retrieval, with natural-language tool-use policy constraints. It builds on BIRD-SQL to create over 7,000 executable tools across 62 domains, extends them with retrieval indices, and constructs tasks in three settings: API-style interaction diversity, multi-hop API reasoning, and multi-source reasoning with policies. Correctness is evaluated by a three-stage waterfall: re-executing predicted tool calls against live APIs with a programmatic containment check and an LLM fallback, judging final-answer groundedness and correctness with GPT-OSS-120B, and deterministically checking policy adherence. Using a fixed ReAct harness, the authors report that the best model reaches 70.4% on endpoint-style tasks, drops to 50-51% on compositional APIs, loses over 50% accuracy with reasoning depth, and performs as low as 2.4% on policy-constrained unanswerable queries.","tokens_in":30113,"tokens_out":8497,"duration_ms":67959,"significance":"If the evaluation methodology is sound, VAKRA would be a valuable, reusable benchmark that combines API grounding, cross-source compositional reasoning, and policy adherence in a single executable environment. The live re-execution of tool calls, checksum-verified tool surfaces, accommodation of multiple valid execution paths, and the fixed ReAct harness are genuine strengths that address reproducibility and architecture confounds. The paper also contributes a detailed data-generation pipeline and an initial human quality study of the generated questions. However, the central accuracy numbers and the trace-level conclusions depend on an unvalidated LLM judge for final-answer scoring and for part of the trajectory verification; until that judge is validated against human annotations on VAKRA's own task distribution, the quantitative findings cannot be considered established.","major_comments":[{"comment":"The two-stage waterfall relies on GPT-OSS-120B as an LLM judge, but no human agreement, calibration, or error analysis is reported for VAKRA's own judge decisions, and no statistics are given for how often Stage 1 is resolved by the programmatic containment check versus the LLM fallback. Because Stage 2 is gated by Stage 1 and the judge determines the 'Gnd' column in Table 4 and all cell values in Table 3, an unvalidated judge makes the reported accuracy numbers and the Section 5.3 trace-level conclusions (e.g., failures concentrate in language-mediated reasoning) unverifiable. Moreover, GPT-OSS-120B is itself one of the evaluated models in Table 3, so self-preference bias is possible and unaddressed. I recommend reporting a human agreement study on a stratified sample of VAKRA trajectories, with separate agreement for Stage 1 and Stage 2, a confusion matrix for the judge, and the frequency of programmatic versus LLM resolution in Stage 1.","section":"Section 4 (Evaluation), Stage 1 and Stage 2"},{"comment":"The main text reports inter-annotator agreement of 77% and 90%, but Table 11 shows Cohen's kappa near zero or negative on most rubric dimensions (e.g., Faithfulness kappa = -0.088 for Multi-hop and -0.023 for Multi-source; Logical Consistency = 0.014 and -0.039). Exact agreement without chance correction is not evidence of reliable annotation, so the high-quality threshold (average score >= 3.0) and the derived percentages (87% and 96% high-quality) may be driven by chance agreement. Please report kappa or Krippendorff's alpha in the main text, and either substantiate the data-quality conclusion with a more appropriate reliability measure or soften the claim.","section":"Section 3.2 and Appendix D, Table 11"},{"comment":"Claude-Opus-4.7 was evaluated on a subset 'due to cost considerations', but no subset size or selection criteria are given, and its policy-category percentage (2.4% for unanswerable queries) is used in the headline finding that policy-constrained questions expose severe failures. Comparing this model's percentages with those of models evaluated on the full test set is not interpretable without knowing the sample size and selection criteria. Please report the number of queries per model per setting and per policy category, or exclude subset-evaluated models from cross-model comparisons and headline claims.","section":"Table 3 and Section 5.2"}],"minor_comments":[{"comment":"The abstract and Section 1 state 'over 8,000 executable APIs', but Section C.5 and Figure 5 report 7,087 tools total across the 62 domains; please reconcile these numbers.","section":"Abstract and Section C.5"},{"comment":"The table is typeset without column separators in the provided text (e.g., 'GPT-5.551.0 50.04' and '50.13.7'), and the main text refers to 'the final three columns' while four policy-category percentages appear; please fix the table formatting and define each column, including the exact computation of 'Avg. Score' with the stated double-weighting rule.","section":"Table 3"},{"comment":"Figures 2 and 3 do not show error bars or per-hop sample counts; please add sample sizes and, if models are run at nonzero sampling temperature, confidence intervals or multiple-seed means so the model-ranking and degradation claims can be assessed.","section":"Section 5.3, Figures 2 and 3"},{"comment":"The text refers to 'Appendix 9' but the actual semantic cluster table appears to be Table 9 in Appendix C; please correct the cross-reference.","section":"Section 3.2"},{"comment":"The API answerability check prompt asks the judge about writing SQL and refers to an 'SQL assistant', while the benchmark evaluates API tool calls; please align the prompt terminology with the actual benchmark setting.","section":"Appendix J.7.4"},{"comment":"There are minor typographical inconsistencies that should be cleaned up: 'multihop' vs 'multi-hop', 'multihp' in the Table 8 caption, 'LLaMA-3.3-70B-Instrust' in Table 12, and inconsistent naming of 'Mistral-Large-3-67B' vs 'Mistral-Large-3-675B' in Table 6.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The judge-validation gap described in the major comments is the main blocker to accepting the reported numbers. The benchmark infrastructure—live re-execution, checksum verification, a fixed harness, and open code/data—is solid and the authors are well placed to add the required validation. The related-work comparison appears fair, and I see no integrity or citation concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"VAKRA is worth engaging with. The benchmark itself is a real contribution: it combines locally hosted executable APIs, document retrieval, multi-hop chains, and natural-language tool-use policies in one environment, with trajectory-level verification by re-executing tool calls against live databases. The related-work table is credible; I don't know of a prior benchmark that does all of this jointly. Releasing code and data, plus a human study on query quality, counts in its favor. The fixed ReAct harness and checksum-verified tool surface make the environment reproducible. The reliance on LiveAPIBench is fine, since the extension is substantial.\n\nThe main empirical pattern—models drop sharply with reasoning depth and do poorly on policy-constrained unanswerable queries—is plausible and consistent with other agent evaluations. I'd trust the relative ordering within a setting more than the absolute scores.\n\nThe soft spot is the two-stage LLM judge, where the stress-test note lands. Stage 1 falls back to GPT-OSS-120B for semantic equivalence over sets of tool responses; Stage 2 uses the same model for groundedness and answer correctness. GPT-OSS-120B is also one of the evaluated models, so self-preference bias is possible. No human agreement, no per-stage error analysis, and no breakdown of how often Stage 1 is programmatic versus judge-mediated. That matters because every number in Table 3 depends on this waterfall. This is a validation gap, not an integrity problem, and it is addressable: a few hundred human judgments on VAKRA's own decisions would settle it.\n\nA few smaller issues: the abstract says over 8,000 APIs while Appendix C.5 says 7,087 tools—one of those numbers is wrong. The main text says 2–5 hop chains but Appendix I.2 samples hop counts from {1,2,3}; that needs alignment. Claude Opus-4.7 is evaluated on a subset and then appears in the same table as full models; that comparison needs a caveat or a different presentation. And there are no confidence intervals or significance tests on model differences; with judge variance in the loop, that is a real omission, though the headline findings would probably survive with error bars.\n\nBottom line: this paper deserves a serious referee. It gives the agent community a usable measurement tool, and the engineering is solid. I would want the judge validated before trusting the absolute scores, but the benchmark contribution stands on its own.","headline":"A genuinely reusable benchmark that combines executable APIs, retrieval, multi-hop chains, and policy constraints, but the headline scores need a judge-validation study before they can be read as capability measurements.","tokens_in":30688,"tokens_out":3297,"would_cite":true,"duration_ms":26592,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"VAKRA spans 8,000+ executable APIs across 62 domains, requiring agents to chain API calls, retrieval, and tool-use policies in one trajectory; the best model scores 70.4% on the easiest tasks and 2.4% when a policy makes a query…","keywords":["VAKRA","agent benchmark","multi-hop reasoning","API tool use","retrieval-augmented generation","tool-use policy adherence","LLM-as-judge","ReAct agents"],"falsifier":"Have three human annotators re-apply the same three-stage waterfall to a stratified sample of roughly 200 trajectories spanning all settings, and measure agreement with the LLM judge's pass/fail decisions on Stage 1 and Stage 2; if agreement is low, or if disagreements concentrate in the Stage 1 containment calls the judge was invoked for, the reported accuracies and the conclusion that failures are language-mediated rather than tool-mechanical would not be established at their stated magnitude.","tokens_in":29652,"feed_emoji":"🧩","tokens_out":14638,"duration_ms":104197,"temperature":0.7,"pith_summary":"Enterprise agents must reason across structured APIs and document collections, but existing benchmarks test these abilities in isolation. VAKRA is a benchmark that combines them in a single reasoning chain — nested API calls, retrieval from document collections, and natural-language tool-use policies — with correctness checked by re-executing predicted tool calls against live database-backed APIs. The empirical finding is that current models are well short of deployment-ready performance on this combination: the strongest model tested reaches 70.4% on the simplest endpoint-style tasks, drops to 50–51% on compositional business-intelligence APIs, loses more than half its accuracy as chain depth grows, and scores 2.4% when a policy makes a question unanswerable. Trace-level analysis attributes most failures to language-mediated steps such as entity disambiguation and cross-source grounding rather than to the mechanics of invoking a tool. If the benchmark's measurements hold, they imply that progress demands better compositional reasoning and constraint interpretation, not just better tool-calling interfaces.","feed_headline":"Best agent model scores only 70.4% on new benchmark","feed_subtitle":"VAKRA chains API calls, retrieval, and policy rules in one task; the best model drops to 2.4% on unanswerable queries.","key_machinery":"The load-bearing mechanism is the evaluation waterfall combined with the chain-construction pipeline. Trajectories are scored in three gated stages: (1) tool-sequence verification, which re-executes each predicted call against live database-backed APIs and applies a programmatic containment check — supplemented by an LLM judge adapted from the CRAG framework — to decide whether all ground-truth information was recovered even when the path differs; (2) final-response evaluation by an LLM judge for groundedness and factual correctness; and (3) a deterministic policy-adherence check that no disallowed source was consulted. Multi-hop difficulty is manufactured by a construction pipeline that links BIRD-SQL queries into a query connectivity graph — edges connect queries whose answer entities parameterize other queries' inputs — then traverses it depth-first with hop counts weighted toward two hops, merges per-hop questions into single queries with an LLM, and adds retrieval edges grounded in Wikipedia passages only after answerability filtering ensures API and retrieval sources cannot answer each other's questions. The fixed ReAct harness wraps every model identically, so the measured differences are attributed to the models' reasoning rather than the scaffolding.","core_discovery":"The paper's central claim is that VAKRA is the first benchmark to require, within a single verified reasoning chain, nested API sequences, cross-source grounding between structured databases and unstructured documents, and adherence to natural-language tool-use policies — and that its measured scores accurately reflect current model capability on that combination. To support this, the benchmark provides more than 8,000 executable API tools across 62 domains derived from real database schemas, paired with domain-aligned document collections and human-validated multi-hop questions, and verifies predictions by re-executing each predicted tool call against the live environment so that multiple valid solution paths can pass. The headline empirical result is a capability ceiling well below deployment readiness: the best model (GPT-5.5) achieves 70.4% on single-hop endpoint tasks and 50–51% on compositional BI APIs, most models lose over 50% accuracy when reasoning chains lengthen, and policy-constrained unanswerable queries are answered correctly only 2.4% of the time by the worst-affected frontier model. A sieve analysis of where tool calls fail (tool selection, argument names, argument values, final grounding) shows the bottleneck is language-mediated reasoning such as entity disambiguation, schema alignment, and cross-source grounding, with hallucinations dominating the remaining grounding errors.","pith_inferences":["If the 2.4% unanswerable-query result reflects a general tendency rather than a quirk of this benchmark's phrasing, then adding an explicit answerability gate — prompting or training models to check whether any permitted tool could actually supply the answer before they respond — is a cheap, testable intervention that could yield the largest single accuracy gain on policy-constrained deployments.","The paper's evidence that failures concentrate at language-mediated steps is correlational; a causal test would insert deterministic entity-linking and schema-mapping modules between hops and measure whether multi-hop accuracy rises more than it does with better tool selection.","The benchmark's construction produces hop-level labels as a byproduct; those labels could support hop-level supervision, such as fine-tuning models on intermediate grounding steps, an extension the released data enables but the paper does not explore.","Because every model ran in the same minimal ReAct harness, the reported ceiling is a statement about that harness, not about the models' absolute potential: a planner-augmented agent on the same tasks could plausibly score higher without any change to the underlying models."],"forward_implications":["A single tool-calling score does not predict capability across interface styles: model rankings invert between endpoint-style Dashboard APIs and compositional business-intelligence APIs, so benchmark suites need multiple interaction paradigms to compare agents fairly.","Reasoning depth itself is a major cost: most models lose over half their accuracy going from one hop to two or more, and hops that end with retrieval followed by an API push grounding and extraction error rates to 70–75% on large models, so depth should be reported separately in agent evaluations.","Policy adherence is a distinct failure mode that final-answer accuracy hides: when a policy renders a question unanswerable, the correct behavior is to refuse, yet models instead force answers (as low as 2.4%), so deployments that require refusal behavior cannot rely on today's agents.","Trajectory-level re-execution scoring lets a benchmark distinguish 'chose the wrong tool' from 'chose the right tool but grounded the arguments badly'; the sieve columns (Tool, ArgN, ArgV, Gnd) show these failure types have different error profiles for SEL versus SLOT interfaces.","Because the benchmark is self-hosted with checksum-verified tool surfaces and a one-command Docker setup, other groups can reproduce the exact tool environment, making the reported numbers a stable comparison point for future models."],"supporting_citations":[{"why":"Supplies the API-generation pipeline that turns BIRD-SQL databases into over 8,000 executable tools, which VAKRA extends with documents, policies, and alternate-trace verification.","marker":"(Elder et al., 2026)"},{"why":"BIRD-SQL: the source of the real database queries and 62 domains from which every API tool and task is derived.","marker":"(Li et al., 2023a)"},{"why":"MuSiQue: provides the connected-reasoning question-composition method (query connectivity graph, depth-first traversal) used to build multi-hop chains.","marker":"(Trivedi et al., 2022)"},{"why":"ReAct: the fixed reason–act–observe harness that every model is evaluated through, the mechanism used to isolate model capability from scaffolding.","marker":"(Yao et al., 2023)"},{"why":"τ-bench: the closest prior benchmark in spirit on tool-use policy adherence, against which VAKRA's multi-domain cross-source policy setting is defined.","marker":"(Yao et al., 2025)"},{"why":"CRAG: supplies the LLM-judge prompt adapted for Stage 1 trajectory-containment verification.","marker":"(Yang et al., 2024)"},{"why":"RAGAS: provides the answer-correctness and groundedness framework used by the Stage 2 final-response judge.","marker":"(Es et al., 2024)"},{"why":"ClapNQ: the document collection that populates the domain-aligned retrieval indices and RAG-API queries.","marker":"(Rosenthal et al., 2025)"},{"why":"Wikidata5M: provides entity identifiers and triples for knowledge-graph construction and cross-source grounding.","marker":"(Wang et al., 2021)"}],"fun_headline_variants":["Best AI agent scores 70.4% on single-hop tasks, 2.4% on unanswerable ones","New benchmark exposes AI weakness: 50% accuracy loss with deeper reasoning","AI agents fail 97.6% on unanswerable queries in new multi-hop benchmark","VAKRA: multi-hop API reasoning drops to 50-51% for best model"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Every reported score passes through a two-stage LLM judge that decides whether an agent's tool-call trajectory and final answer are correct, and the paper assumes that judge is reliable without reporting how often it agrees with human judgment on VAKRA's own tasks.","fun_headline_variants_meta":{"raw":{"variants":["Best AI agent scores 70.4% on single-hop tasks, 2.4% on unanswerable ones","New benchmark exposes AI weakness: 50% accuracy loss with deeper reasoning","AI agents fail 97.6% on unanswerable queries in new multi-hop benchmark","VAKRA: multi-hop API reasoning drops to 50-51% for best model"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001429,"raw_usage":{"total_tokens":5828,"prompt_tokens":1076,"completion_tokens":4752,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":692,"completion_tokens_details":{"reasoning_tokens":4655}},"tokens_in":692,"tokens_out":4752,"duration_ms":29547,"temperature":1.0,"reasoning_tokens":4655,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:09:48.888014+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have three human annotators re-apply the same three-stage waterfall to a stratified sample of roughly 200 trajectories spanning all settings, and measure agreement with the LLM judge's pass/fail decisions on Stage 1 and Stage 2; if agreement is low, or if disagreements concentrate in the Stage 1 containment calls the judge was invoked for, the reported accuracies and the conclusion that failures are language-mediated rather than tool-mechanical would not be established at their stated magnitude.","supporting_citations":[{"cited_title":"So, any query which could be connected via an retriever question was only used for the pur- pose of constructing API-RAG style joint rea- soning queries","cited_arxiv_id":null,"evidence_quote":"Wikidata5M: provides entity identifiers and triples for knowledge-graph construction and cross-source grounding."}],"review_version":1}