Pith. sign in

REVIEW 3 major objections 6 minor 52 references

VAKRA: Evaluating Multi-Hop Reasoning Across APIs and Retrieval Under Tool-Use Policies

T0 review · 3 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read VAKRA spans 8,000+ executable APIs across 62 domains, requiring agents to chain API calls, retrieval, and tool-use policies in one trajectory; the best model scores 70.4% on the easiest tasks and 2.4% when a policy makes a query…

desk verdict A genuinely reusable benchmark that combines executable APIs, retrieval, multi-hop chains, and policy constraints, but the headline scores need a judge-validation study before they can be read as capability measurements. read the letter →

arxiv 2608.12282 v1 pith:IYWWMQQC submitted 2026-08-12 cs.AI

Ankita Rajaram Naik , Anupama Murthi , Benjamin Elder , Siyu Huo , Raavi Gupta , Abhinav Jain , Praveen Venkateswaran , Abdulhamid Adebayo
show 1 more author
Danish Contractor
This is my paper · ORCID
classification cs.AI
keywords VAKRAagentbenchmarkmulti-hopreasoningAPItooluseretrieval-augmentedgenerationtool-usepolicyadherenceLLM-as-judgeReActagents
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Enterprise agents must reason across structured APIs and document collections, but existing benchmarks test these abilities in isolation. VAKRA is a benchmark that combines them in a single reasoning chain — nested API calls, retrieval from document collections, and natural-language tool-use policies — with correctness checked by re-executing predicted tool calls against live database-backed APIs. The empirical finding is that current models are well short of deployment-ready performance on this combination: the strongest model tested reaches 70.4% on the simplest endpoint-style tasks, drops to 50–51% on compositional business-intelligence APIs, loses more than half its accuracy as chain depth grows, and scores 2.4% when a policy makes a question unanswerable. Trace-level analysis attributes most failures to language-mediated steps such as entity disambiguation and cross-source grounding rather than to the mechanics of invoking a tool. If the benchmark's measurements hold, they imply that progress demands better compositional reasoning and constraint interpretation, not just better tool-calling interfaces.

What carries the argument

The load-bearing mechanism is the evaluation waterfall combined with the chain-construction pipeline. Trajectories are scored in three gated stages: (1) tool-sequence verification, which re-executes each predicted call against live database-backed APIs and applies a programmatic containment check — supplemented by an LLM judge adapted from the CRAG framework — to decide whether all ground-truth information was recovered even when the path differs; (2) final-response evaluation by an LLM judge for groundedness and factual correctness; and (3) a deterministic policy-adherence check that no disallowed source was consulted. Multi-hop difficulty is manufactured by a construction pipeline that links BIRD-SQL queries into a query connectivity graph — edges connect queries whose answer entities parameterize other queries' inputs — then traverses it depth-first with hop counts weighted toward two hops, merges per-hop questions into single queries with an LLM, and adds retrieval edges grounded in Wikipedia passages only after answerability filtering ensures API and retrieval sources cannot answer each other's questions. The fixed ReAct harness wraps every model identically, so the measured differences are attributed to the models' reasoning rather than the scaffolding.

What would settle it

Have three human annotators re-apply the same three-stage waterfall to a stratified sample of roughly 200 trajectories spanning all settings, and measure agreement with the LLM judge's pass/fail decisions on Stage 1 and Stage 2; if agreement is low, or if disagreements concentrate in the Stage 1 containment calls the judge was invoked for, the reported accuracies and the conclusion that failures are language-mediated rather than tool-mechanical would not be established at their stated magnitude.

Watch

Extended reading notes

Core claim

The paper's central claim is that VAKRA is the first benchmark to require, within a single verified reasoning chain, nested API sequences, cross-source grounding between structured databases and unstructured documents, and adherence to natural-language tool-use policies — and that its measured scores accurately reflect current model capability on that combination. To support this, the benchmark provides more than 8,000 executable API tools across 62 domains derived from real database schemas, paired with domain-aligned document collections and human-validated multi-hop questions, and verifies predictions by re-executing each predicted tool call against the live environment so that multiple valid solution paths can pass. The headline empirical result is a capability ceiling well below deployment readiness: the best model (GPT-5.5) achieves 70.4% on single-hop endpoint tasks and 50–51% on compositional BI APIs, most models lose over 50% accuracy when reasoning chains lengthen, and policy-constrained unanswerable queries are answered correctly only 2.4% of the time by the worst-affected frontier model. A sieve analysis of where tool calls fail (tool selection, argument names, argument values, final grounding) shows the bottleneck is language-mediated reasoning such as entity disambiguation, schema alignment, and cross-source grounding, with hallucinations dominating the remaining grounding errors.

Load-bearing premise

Every reported score passes through a two-stage LLM judge that decides whether an agent's tool-call trajectory and final answer are correct, and the paper assumes that judge is reliable without reporting how often it agrees with human judgment on VAKRA's own tasks.

Editorial extensions

If this is right

  • A single tool-calling score does not predict capability across interface styles: model rankings invert between endpoint-style Dashboard APIs and compositional business-intelligence APIs, so benchmark suites need multiple interaction paradigms to compare agents fairly.
  • Reasoning depth itself is a major cost: most models lose over half their accuracy going from one hop to two or more, and hops that end with retrieval followed by an API push grounding and extraction error rates to 70–75% on large models, so depth should be reported separately in agent evaluations.
  • Policy adherence is a distinct failure mode that final-answer accuracy hides: when a policy renders a question unanswerable, the correct behavior is to refuse, yet models instead force answers (as low as 2.4%), so deployments that require refusal behavior cannot rely on today's agents.
  • Trajectory-level re-execution scoring lets a benchmark distinguish 'chose the wrong tool' from 'chose the right tool but grounded the arguments badly'; the sieve columns (Tool, ArgN, ArgV, Gnd) show these failure types have different error profiles for SEL versus SLOT interfaces.
  • Because the benchmark is self-hosted with checksum-verified tool surfaces and a one-command Docker setup, other groups can reproduce the exact tool environment, making the reported numbers a stable comparison point for future models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the 2.4% unanswerable-query result reflects a general tendency rather than a quirk of this benchmark's phrasing, then adding an explicit answerability gate — prompting or training models to check whether any permitted tool could actually supply the answer before they respond — is a cheap, testable intervention that could yield the largest single accuracy gain on policy-constrained deployments.
  • The paper's evidence that failures concentrate at language-mediated steps is correlational; a causal test would insert deterministic entity-linking and schema-mapping modules between hops and measure whether multi-hop accuracy rises more than it does with better tool selection.
  • The benchmark's construction produces hop-level labels as a byproduct; those labels could support hop-level supervision, such as fine-tuning models on intermediate grounding steps, an extension the released data enables but the paper does not explore.
  • Because every model ran in the same minimal ReAct harness, the reported ceiling is a statement about that harness, not about the models' absolute potential: a planner-augmented agent on the same tasks could plausibly score higher without any change to the underlying models.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces VAKRA, a benchmark for evaluating multi-hop agentic reasoning across structured APIs and document retrieval, with natural-language tool-use policy constraints. It builds on BIRD-SQL to create over 7,000 executable tools across 62 domains, extends them with retrieval indices, and constructs tasks in three settings: API-style interaction diversity, multi-hop API reasoning, and multi-source reasoning with policies. Correctness is evaluated by a three-stage waterfall: re-executing predicted tool calls against live APIs with a programmatic containment check and an LLM fallback, judging final-answer groundedness and correctness with GPT-OSS-120B, and deterministically checking policy adherence. Using a fixed ReAct harness, the authors report that the best model reaches 70.4% on endpoint-style tasks, drops to 50-51% on compositional APIs, loses over 50% accuracy with reasoning depth, and performs as low as 2.4% on policy-constrained unanswerable queries.

Significance. If the evaluation methodology is sound, VAKRA would be a valuable, reusable benchmark that combines API grounding, cross-source compositional reasoning, and policy adherence in a single executable environment. The live re-execution of tool calls, checksum-verified tool surfaces, accommodation of multiple valid execution paths, and the fixed ReAct harness are genuine strengths that address reproducibility and architecture confounds. The paper also contributes a detailed data-generation pipeline and an initial human quality study of the generated questions. However, the central accuracy numbers and the trace-level conclusions depend on an unvalidated LLM judge for final-answer scoring and for part of the trajectory verification; until that judge is validated against human annotations on VAKRA's own task distribution, the quantitative findings cannot be considered established.

major comments (3)
  1. [Section 4 (Evaluation), Stage 1 and Stage 2] The two-stage waterfall relies on GPT-OSS-120B as an LLM judge, but no human agreement, calibration, or error analysis is reported for VAKRA's own judge decisions, and no statistics are given for how often Stage 1 is resolved by the programmatic containment check versus the LLM fallback. Because Stage 2 is gated by Stage 1 and the judge determines the 'Gnd' column in Table 4 and all cell values in Table 3, an unvalidated judge makes the reported accuracy numbers and the Section 5.3 trace-level conclusions (e.g., failures concentrate in language-mediated reasoning) unverifiable. Moreover, GPT-OSS-120B is itself one of the evaluated models in Table 3, so self-preference bias is possible and unaddressed. I recommend reporting a human agreement study on a stratified sample of VAKRA trajectories, with separate agreement for Stage 1 and Stage 2, a confusion matrix for the judge, and the frequency of programmatic versus LLM resolution in Stage 1.
  2. [Section 3.2 and Appendix D, Table 11] The main text reports inter-annotator agreement of 77% and 90%, but Table 11 shows Cohen's kappa near zero or negative on most rubric dimensions (e.g., Faithfulness kappa = -0.088 for Multi-hop and -0.023 for Multi-source; Logical Consistency = 0.014 and -0.039). Exact agreement without chance correction is not evidence of reliable annotation, so the high-quality threshold (average score >= 3.0) and the derived percentages (87% and 96% high-quality) may be driven by chance agreement. Please report kappa or Krippendorff's alpha in the main text, and either substantiate the data-quality conclusion with a more appropriate reliability measure or soften the claim.
  3. [Table 3 and Section 5.2] Claude-Opus-4.7 was evaluated on a subset 'due to cost considerations', but no subset size or selection criteria are given, and its policy-category percentage (2.4% for unanswerable queries) is used in the headline finding that policy-constrained questions expose severe failures. Comparing this model's percentages with those of models evaluated on the full test set is not interpretable without knowing the sample size and selection criteria. Please report the number of queries per model per setting and per policy category, or exclude subset-evaluated models from cross-model comparisons and headline claims.
minor comments (6)
  1. [Abstract and Section C.5] The abstract and Section 1 state 'over 8,000 executable APIs', but Section C.5 and Figure 5 report 7,087 tools total across the 62 domains; please reconcile these numbers.
  2. [Table 3] The table is typeset without column separators in the provided text (e.g., 'GPT-5.551.0 50.04' and '50.13.7'), and the main text refers to 'the final three columns' while four policy-category percentages appear; please fix the table formatting and define each column, including the exact computation of 'Avg. Score' with the stated double-weighting rule.
  3. [Section 5.3, Figures 2 and 3] Figures 2 and 3 do not show error bars or per-hop sample counts; please add sample sizes and, if models are run at nonzero sampling temperature, confidence intervals or multiple-seed means so the model-ranking and degradation claims can be assessed.
  4. [Section 3.2] The text refers to 'Appendix 9' but the actual semantic cluster table appears to be Table 9 in Appendix C; please correct the cross-reference.
  5. [Appendix J.7.4] The API answerability check prompt asks the judge about writing SQL and refers to an 'SQL assistant', while the benchmark evaluates API tool calls; please align the prompt terminology with the actual benchmark setting.
  6. [Throughout] There are minor typographical inconsistencies that should be cleaned up: 'multihop' vs 'multi-hop', 'multihp' in the Table 8 caption, 'LLaMA-3.3-70B-Instrust' in Table 12, and inconsistent naming of 'Mistral-Large-3-67B' vs 'Mistral-Large-3-675B' in Table 6.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: benchmark scores are live-execution measurements; judge self-overlap is a validation risk, not a definitional reduction.

full rationale

VAKRA's claimed contributions are benchmark construction and observed model measurements, not derived predictions. No equation or scoring rule in the paper fits a parameter and then evaluates that parameter on the same quantity. Tool-sequence correctness is checked by re-executing predicted calls against self-hosted, database-backed APIs with a programmatic containment check and an LLM fallback only for inconclusive cases (Section 4, Stage 1); policy adherence is checked deterministically against per-query source tags (Section 4, Stage 3); and final-answer correctness is judged for groundedness in tool responses using RAGAS-style prompts. The multi-hop tasks are generated from BIRD-SQL via a query connectivity graph and are independently vetted by a three-annotator human study (Section 3.2, Appendix D), which is external evidence about query quality rather than a circular validation. The harness is a fixed, uniform ReAct loop, so model scores measure the model rather than the benchmark's construction. The one notable risk—using GPT-OSS-120B as the LLM judge while also reporting GPT-OSS-120B's accuracy—is a validation and self-preference concern that could change the reported numbers, but it is not a circular reduction: no score is defined as the judge's output by construction, and the same judge is applied to all models. Building on the authors' LiveAPIBench pipeline is a normal dependency, not a load-bearing self-citation that forces the central results; the new benchmark content and the live re-execution evaluation are independent of it. Finding: no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. Its free parameters are design choices in the benchmark generation and scoring pipeline, and its axioms are domain assumptions about the validity of LLM-generated tasks, LLM-based filtering, and LLM-as-judge scoring. These assumptions are load-bearing for the reported numbers, which is why they are listed explicitly.

free parameters (4)
  • hop count sampling weights = 0.10 / 0.60 / 0.30 for 1 / 2 / 3 hops
    Chosen by hand in Section I.2 (Algorithm 1); this shapes the difficulty distribution of the multi-hop dataset and therefore the reported accuracy-by-hop curves.
  • multi-source scoring weight = 2
    Section 4 states that multi-source queries receive double weight relative to single-source queries when computing the average score; this directly affects the Table 3 averages.
  • document corpus size per domain = 20,000
    Section I.5 samples surviving documents to 20,000 per domain for the ChromaDB index; this changes retrieval difficulty and the impact of answerability filtering.
  • human quality threshold = 3.0 on a 1-4 scale
    Section 3.2 defines a sample as high-quality if its average rubric score is at least 3.0; the reported 87% and 96% quality rates depend on this cut.
assumptions (4)
  • domain assumption BIRD-SQL question-SQL pairs, converted to executable APIs via the LiveAPIBench pipeline, provide a valid stand-in for real enterprise API tool use.
    The entire task universe is generated from BIRD-SQL, Wikidata, and ClapNQ; external validity of the benchmark depends on this mapping, stated in Sections 3 and 3.1.
  • domain assumption The LLM-based answerability filtering and cross-source contamination checks in the generation pipeline are accurate enough to prevent shortcut answers.
    Sections I.4, I.5, and J.4 rely on Mistral and Mixtral judges to remove RAG questions answerable via APIs and vice versa; no independent validation of filter accuracy is reported.
  • domain assumption GPT-OSS-120B is a reliable judge for trajectory containment and final-answer correctness.
    Section 4 uses this model at temperature 0 for Stages 1 and 2 without reporting validation against human labels; all headline accuracy numbers depend on it.
  • domain assumption Deterministic SQLite-backed functions and ChromaDB semantic search adequately represent real API and document retrieval environments.
    Section 4.1 describes the self-hosted environment; benchmark realism depends on this representation, although it enables deterministic verification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VAKRA: Evaluating Multi-Hop Reasoning Across APIs and Retrieval Under Tool-Use Policies." pith.science (2026). https://pith.science/paper/IYWWMQQC

@misc{pith2026260812282,
  author       = {Pith},
  title        = {Pith review of: VAKRA: Evaluating Multi-Hop Reasoning Across APIs and Retrieval Under Tool-Use Policies},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IYWWMQQC}},
  note         = {Machine review of arXiv:2608.12282}
}
abstract

Agents deployed in enterprise settings must reason across structured APIs and document collections, yet existing benchmarks evaluate these capabilities in isolation. We introduce VAKRA (e\textbf{V}aluating \textbf{A}PI and \textbf{K}nowledge \textbf{R}etrieval \textbf{A}gents), a benchmark of over $8{,}000$ executable APIs across $62$ domains with tasks spanning three settings of increasing difficulty: diverse API interaction styles, multi-hop reasoning over structured APIs, and multi-source reasoning with natural-language tool-use policy constraints. Correctness is verified by re-executing predicted tool calls against live APIs, accommodating multiple valid paths. Using a fixed ReAct harness to isolate model capabilities from agent architecture, we evaluate frontier and open-weight models and find that even the best model achieves only 70.4\% on single-hop endpoint-style tasks and drops to 50--51\% on compositional APIs; performance degrades by over 50\% as reasoning depth increases, and policy-constrained questions expose severe failures (as low as 2.4\% on unanswerable queries). Trace analysis shows failures concentrate at language-mediated reasoning - entity disambiguation, cross-source grounding, rather than tool invocation mechanics. Code is available https://github.com/IBM/VAKRA. Dataset is available https://huggingface.co/datasets/ibm-research/VAKRA

Figures

Figures reproduced from arXiv: 2608.12282 by the authors.

Figure 1
Figure 1. Example enterprise workflow illustrating reasoning challenges including (i) [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Model Accuracy Rates by Interaction Types [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Model Accuracy by number of hops for Mul [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Question Type Distribution for Multihop setting and the MultiHop Multisource questions [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Distribution of the number of tools per domain [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Data sample demonstrating a sample for each of the settings (i) SEL Business Intelligence API type (ii) [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 48 canonical work pages

  1. [1]

    So, any query which could be connected via an retriever question was only used for the pur- pose of constructing API-RAG style joint rea- soning queries

    Only a limited number of queries had entities which could have linkages with base Wiki- data5m passages (Wang et al., 2021) required to form a knowledge graph needed for Mul- tihop and Multisource query generation. So, any query which could be connected via an retriever question was only used for the pur- pose of constructing API-RAG style joint rea- soni...

    work page 2021

  2. [2]

    InFindings of the Association for Computational Linguistics: ACL 2025, pages 5880–5895

    An empirical study of llm-as-a-judge for llm evaluation: Fine-tuned judge model is not a general substitute for gpt-4. InFindings of the Association for Computational Linguistics: ACL 2025, pages 5880–5895. Yannis Katsis, Sara Rosenthal, Kshitij Fadnis, Chu- laka Gunasekara, Young-Suk Lee, Lucian Popa, Vraj Shah, Huaiyu Zhu, Danish Contractor, and Marina ...

    work page 2025

  3. [3]

    All the other queries which couldn’t be added to the knowledge graphs due to entities which couldn’t be linked to other queries or their answers were used as the queries for the Dash- board APIs task

    work page

  4. [4]

    InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), pages 16022–16076, Bangkok, Thai- land

    AppWorld: A controllable world of apps and people for benchmarking interactive coding agents. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), pages 16022–16076, Bangkok, Thai- land. Association for Computational Linguistics. Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan...

    work page 2021

  5. [5]

    arXiv preprint arXiv:2508.20453

    Mcp-bench: Benchmarking tool-using llm agents with complex real-world tasks via mcp servers. arXiv preprint arXiv:2508.20453. Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang. 2023. On the tool manipulation capability of open-source large lan- guage models.Preprint, arXiv:2305.16504. 10 Xiao Yang, Kai Sun, Hao Xin, Yushi Sun, Nik...

    arXiv 2023

  6. [9]

    Queries which could form (API-API) linkages based on the query entities and answer enti- ties were reserved for multihop setting of the dataset

    work page

  7. [11]

    SEL and SLOT) already leads to a relatively more difficult task the queries used for this split of the dataset could be overlapping with other settings

    As the nesting behaviour of Business Intel- ligence APIs (i.e. SEL and SLOT) already leads to a relatively more difficult task the queries used for this split of the dataset could be overlapping with other settings

    work page

  8. [12]

    Do not use documents to answer questions related to movies

    No component BIRD-SQL are shared be- tween the tuning and test set of our dataset. C.4 Data Sample Figure 6 shows a data sample for SEL Business Intelligence API and the ground truth tool calls re- quired to answer the query as well as a data sample from the Dashboard APIs collection and structure of APIs required to answer queries. It also shows a data s...

    work page 2022

Show all 52 references
  1. [13]

    Following is the grounded- ness prompt used for stage 2 : GroundednessPrompt =""" The following tasks each contains document and a response

    answer correctness prompt is used for factual correctness in stage 2. Following is the grounded- ness prompt used for stage 2 : GroundednessPrompt =""" The following tasks each contains document and a response. The response is supposed to rely on the document for its source of...

  2. [14]

    I'm a helpful assistant

    Identify all substantial claims in the response: - Ignore non-substantial claims, such as greetings or self-descriptions such as "I'm a helpful assistant", - Try to formulate each claim in a stand- alone form with all pronouns and other references resolved

  3. [15]

    Assess the grounding of each of these claims: - If it is essentially a rephrasing of information from the document, or can be derived from such information by trivial common-sense reasoning, it is grounded, This is so even if it contradicts other parts of the document. - If it...

  4. [16]

    yes" is for grounded, -

    Make the overall decision according to: - If at least one claim is not grounded, the response is not grounded (Note that this is not a case of partially grounded); - Otherwise if at least one claim is partially grounded, the response is partially grounded; - Otherwise the resp...

    2022

  5. [17]

    Documents capable of answering any existing API query are identified and removed from the retrieval corpus

  6. [18]

    RAG queries answerable via structured APIs are discarded from the task set

  7. [19]

    Cross-domain contamination is checked: doc- uments that can answer queries from unrelated domains are removed

  8. [20]

    Surviving documents are sampled to 20,000 per domain and indexed using ChromaDB, forming the retrieval tools for document- grounded tasks. J Multi-Turn RAG Benchmark Data Generation Pipeline This section describes our automated pipeline for generating high-quality multi-turn c...

  9. [21]

    Set Elasticsearch index to ClapNQ, configure domain name, description and keywords for domain-specific context retrieval

  10. [22]

    Initialize conversation with domain-specific context fromClapNQ

  11. [23]

    Reset conversation history and metadata J.3.2 Turn Generation Loop Each turn in the conversation follows this work- flow: Step 1: User Query GenerationThe User Agent generates a query based on:

  12. [24]

    Retrieved documents from the domain- specific corpus plus Conversation history from previous turns

  13. [25]

    Step 2: RAG Response GenerationThe RAG Agent:

    Determine query type (entity-based, factoid, etc.) The agent employs sophisticated prompts (see Appendix J.7) and conversation, document context to generate questions including: • Entity questions: Queries whose answer is a specific named entity.(see Appendix J.7.1) • Factoid ...

  14. [26]

    Receives the user query concatenated with retrieved passages

  15. [27]

    Uses the prompt template shown in Appendix J.7.3 Step 3: Multi-hop Query MergingFor creating multi-hop queries, the system can merge sequential question-answer pairs:

    Generates response to the user query. Uses the prompt template shown in Appendix J.7.3 Step 3: Multi-hop Query MergingFor creating multi-hop queries, the system can merge sequential question-answer pairs:

  16. [28]

    Randomly selects merge position (turn 1 or turn 3)

  17. [29]

    Extracts first query, answer, and second query from selected turns

  18. [30]

    Uses LLM to generate merged query that com- bines both questions without mentioning in- termediate answer

  19. [31]

    Validates naturalness of merged query

  20. [32]

    Creates new conversation with merged query replacing original turns with multi- hop patterns: (RAG-RAG)(RAG)(RAG) or (RAG)(RAG)(RAG-RAG) Step 4: Data RecordingUpon successful vali- dation, the system records: • User query and metadata (entity, document, query type) • RAG respo...

  21. [33]

    Groundedness Verification: Verify answers to the queries are factually supported by re- trieved documents using LLM-based valida- tion

  22. [34]

    I can not answer

    Unanswerability Detection: Rejects re- sponses containing "I can not answer" or re- ject answering the question

  23. [35]

    Conversation Rejection: Discards entire con- versation if any quality check fails

  24. [36]

    task_name

    Completeness Check: Ensures conversations reach the target number of turns (default: 6) J.4.2 Post-Generation Decontamination After initial data generation, comprehensive decon- tamination ensures no data leakage: 1.Cross-Domain Filtering: • Identifies documents that can answe...

  25. [37]

    Identify one name entity from the document, and this entity should not be among a serial of entities (people's name) in the documents

  26. [38]

    Create one question whose correct answer is exactly this named entity

  27. [39]

    And can not be other entities

    The answer must be the entity name only, with no extra words, explanations, or punctuation. And can not be other entities

  28. [40]

    This", "That

    The question must be answerable only from the document. You should generate following below format: <question> Generate the question and its answer is exactly the one proper noun entity. And the answer can not be other entities mentioned in the documents. The question should b...

  29. [43]

    Read understand the given user queries and an agent responses in the conversations

  30. [44]

    Please generate output following below format: <question> generate question considering below rules:

    Based on provided documents and conversation history, you should generate a question about the proper noun name entity appeared in the document. Please generate output following below format: <question> generate question considering below rules:

  31. [45]

    The question is asking some fact of the entity itself, instead of its belongings nor other proper noun entity

  32. [46]

    what other

    The question should not start with "what other", please ask more straightforward question without double checking previous context for its answer

  33. [47]

    This", "That

    The question should be concise enough and do not require clarification of "This", "That", "These"... nor other co-reference words. If you can not generate such question, write "I can not generate" </question> J.7.2 Follow-up Turn - Entity Answer Selection Prompt You are a help...

  34. [48]

    Read the document and understand its content

  35. [49]

    Identify the main topic or the key point being discussed in the conversation

  36. [50]

    Read understand the given user queries and responses

  37. [51]

    This", "That

    Based on provided documents and conversation history, you should do below two tasks: task1: try to find one proper noun name entity (names start with capital letter) appear in the documents, but it did not appear in the conversation history. And this entity should not be among...

  38. [52]

    I'm a helpful assistant

    Identify all substantial claims in the response: - Ignore non-substantial claims, such as greetings or self-descriptions such as "I'm a helpful assistant", - Try to formulate each claim in a stand-alone form with all pronouns and other references resolved

  39. [53]

    Assess the grounding of each of these claims: - If it is essentially a rephrasing of information from the document, or can be derived from such information by trivial common-sense reasoning, it is grounded, This is so even if it contradicts other parts of the document. - If it...

  40. [54]

    yes" is for grounded, -

    Make the overall decision according to: - If at least one claim is not grounded, the response is not grounded (Note that this is not a case of partially grounded); - Otherwise if at least one claim is partially grounded, the response is partially grounded; - Otherwise the resp...

  41. [2023]

    Shuyan Zhou, Frank F

    Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in neural information pro- cessing systems, 36:46595–46623. Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Gra- ham N...

    2024 arXiv

  42. [2024]

    Multi-document grounded multi-turn synthetic dialog generation.Preprint, arXiv:2409.11500. Jinyang Li, Binyuan Hui, GE QU, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guo- liang Li, Kevin Chang, Fei Huang, Reynold Ch...

    2024 arXiv

  43. [2025]

    Kinjal Basu, Ibrahim Abdelaziz, Kiran Kate, Mayank Agarwal, Maxwell Crouse, Yara Rizk, Kelsey Brad- ford, Asim Munawar, Sadhana Kumaravel, Saurabh Goyal, Xin Wang, Luis A

    Automated creation and enrichment frame- work for improved invocation of enterprise apis as tools.arXiv preprint arXiv:2509.11626. Kinjal Basu, Ibrahim Abdelaziz, Kiran Kate, Mayank Agarwal, Maxwell Crouse, Yara Rizk, Kelsey Brad- ford, Asim Munawar, Sadhana Kumaravel, Saurabh...

    2025

  44. [2026]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others

    Gecko: A simulation environment with state- ful feedback for refining agent tool calls.Preprint, arXiv:2602.19218. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others

    arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.