Agent-UniRAG: A Trainable Open-Source LLM Agent Framework for Unified Retrieval-Augmented Generation Systems

Hoang Pham; Khac-Hoai Nam Bui; Thuy-Duong Nguyen

REVIEW 4 major objections 8 minor 37 references

Agent-UniRAG: A Trainable Open-Source LLM Agent Framework for Unified Retrieval-Augmented Generation Systems

T0 review · 4 major / 8 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read One 8B agent can handle both simple and multi-hop retrieval-augmented questions in a single loop, and training on synthetic traces makes it competitive with GPT-4.

desk verdict A clean, practical small-model RAG agent with a new synthetic dataset, but the 'competitive with GPT-4' claim rests on a self-referential evaluation and the standard benchmarks lack decontamination and error bars. read the letter →

arxiv 2505.22571 v3 pith:5GXVIBU2 submitted 2025-05-28 cs.CL cs.AIcs.DBcs.IR

Hoang Pham , Thuy-Duong Nguyen , Khac-Hoai Nam Bui This is my paper · ORCID

classification cs.CLcs.AIcs.DBcs.IR

keywords retrieval-augmentedgenerationLLMagentsmulti-hopquestionansweringsingle-hopinstructiontuningsyntheticdatadistillationopen-sourceadaptiveretrieval

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a single LLM agent built on a small open-source model can solve both single-hop and multi-hop retrieval-augmented generation (RAG) queries in one end-to-end loop, removing the need to route queries to separate systems. The authors build Agent-UniRAG on Llama-3-8B, using a ReAct-style loop in which a planning module alternates between generating search queries and deciding when to give a final answer, while an evidence reflector filters retrieved passages for noise. To make an 8B model behave this way, they distill GPT-4 into a synthetic instruction dataset, SynAgent-RAG, built from Wikipedia Vital Articles, and fine-tune the model on the resulting 16,987 conversational traces. Across six standard RAG benchmarks, the fine-tuned agent reports competitive scores against GPT-3.5-based baselines, and on the dataset's own test set it approaches GPT-4's semantic score while using fewer search steps. The point of the paper is that a small, trainable, interpretable agent can stand in for much larger and closed models in unified RAG.

What carries the argument

The load-bearing object is the agent loop, assembled from four modules: a Planning Module that uses the ReAct mechanism to emit Thought/Action/Evidence steps and choose between 'Search' and 'Final Answer'; a Search Tool that turns the planned action into a query against the knowledge base; an Evidence Reflector, inspired by Reflexion, that filters retrieved passages and returns concise, relevant evidence or 'No information found'; and a Working Memory that stores the running trace. The companion mechanism is SynAgent-RAG, a distillation pipeline in which GPT-4 generates questions, extracts evidence, and annotates solution paths from linked Wikipedia passages, and a verification module keeps only annotations whose answers score 4 or 5. Training is a multi-task conversational fine-tune of Llama-3-8B with loss masked to the assistant turns (planning responses, evidence extraction, and final answers), which is what teaches the small model the step-by-step search behavior.

What would settle it

Compare token-level overlap between SynAgent-RAG training passages and the corpora and questions of SQuAD, NQ, TriviaQA, MuSiQue, HotpotQA, and 2WikiMultiHopQA. If any benchmark answers appear in training passages, retrain the agent after removing those passages and check whether its margin over the GPT-3.5 baselines on multi-hop datasets survives; if the margin vanishes, the unified-agent claim is not supported.

Watch

Extended reading notes

Core claim

Agent-UniRAG's central claim is that query complexity does not need to be classified in advance: the same agent loop can decide, step by step, whether one search suffices or several are needed. The loop's planner produces a Thought and a Search action, the search tool retrieves passages from a corpus, and the Evidence Reflector compresses what is relevant and returns 'No information found' when nothing matches. Working memory accumulates the question, thoughts, actions, and evidence, and the agent terminates with a Final Answer when the planner is confident or when a preconfigured search budget is exhausted. The authors report that instruction-tuning Llama-3-8B on SynAgent-RAG — which contains single-hop and multi-hop questions with annotated search trajectories and long-form answers — yields an agent whose Exact Match, F1, and accuracy scores are competitive with or better than Self-RAG, IRCoT, and Adaptive-RAG (run with GPT-3.5), and whose GPT-Score on the long-form test set (4.19) is close to GPT-4 (4.35) and above Llama-3-70B (3.62).

Load-bearing premise

The synthetic training data is built from Wikipedia articles, and the six evaluation benchmarks also draw on Wikipedia, but the paper reports no check for overlap between its training passages and the benchmark corpora or questions, so the reported gains could reflect memorized content rather than learned search behavior.

Editorial extensions

If this is right

A single small open-source model can field mixed-complexity queries in one loop, removing the need for a separate classifier or separate single-hop and multi-hop systems.
The recorded thoughts, search queries, and evidence make the model's answers traceable and interpretable.
Strengthening the retriever (adding a dense reranker over BM25) directly improves agent accuracy, so the agent and the retriever are complementary.
The distilled agent uses fewer average search steps than the larger models it is compared with on the dataset's test set, indicating lower inference cost.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because both the training corpus and the six benchmarks stem from Wikipedia, the reported gains could partly reflect topic overlap rather than the learned reasoning loop; a decontaminated re-run would settle this.
The planner's low step count suggests the model implicitly learns to estimate query complexity, so the loop itself could double as a cheap complexity detector.
The same distillation recipe could train small agents for other tool-use tasks (code execution, structured function calling), but the paper only demonstrates RAG.
Training on oracle retrieval may not prepare the agent for realistic retrieval noise; adding failed or partial retrievals to the synthetic traces could make it more robust.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First, the useful core. The paper assembles known ingredients—ReAct-style planning, a Reflexion-like evidence reflector, LUMOS-style trainable open-source agents—into a single agent loop, and contributes SynAgent-RAG, a 17k-sample synthetic dataset that makes the loop work on Llama-3-8B. That is a concrete, reproducible artifact, and the ablations (Table 2) show the planning and reflector modules carry real weight. If the code and data actually ship, this is the kind of systems paper the community can build on.

The problems are in the evaluation framing. The abstract says 'comparable performances with closed-source and larger open-source LLMs across various RAG benchmarks.' On the six standard benchmarks, the comparison is only against GPT-3.5-based baselines; no GPT-4 or Llama-3-70B appears there. The only GPT-4/Llama-70B comparison is Table 3, run on the SynAgent-RAG test set—which is generated by the same GPT-4 teacher pipeline that produced the training set, and scored with an LLM-based evaluator whose independence from the teacher is never established. That is a self-referential evaluation. It is fine to report, but it does not support the headline claim.

Second, the decontamination concern is real. SynAgent-RAG is drawn from Wikipedia Vital Articles Level 5, and the evaluation corpora are also Wikipedia-derived. The paper splits train/test articles but never checks overlap with the QA benchmarks. Without that check, part of the gain on standard datasets could be memorization. The Limitations section is honest about scope and inference cost but silent on this.

Third, 500-sample evaluations have no error bars, and the short-answer extraction for the proposed model uses GPT-4 while the baseline numbers are borrowed from prior papers with a possibly different protocol. That is a moderate fairness issue, but fixable.

Who this is for: anyone building small-model RAG agents. The framework holds up and the dataset is a contribution, but the empirical claims need re-framing and more careful evaluation. A serious editor should send this to peer review; a good reviewer will ask for decontamination checks, a non-circular head-to-head with a large model, and error bars.

Referee Report

4 major / 8 minor

Summary. The paper proposes Agent-UniRAG, a unified LLM-agent framework for retrieval-augmented generation that handles both single-hop and multi-hop queries in an end-to-end loop, and introduces SynAgent-RAG, a synthetic dataset distilled from GPT-4 to train a Llama-3-8B backbone. The authors fine-tune three sub-tasks (planning, evidence extraction, final answer generation) and evaluate the resulting model on six standard short-form RAG benchmarks (SQuAD, Natural Questions, TriviaQA, MuSiQue, HotpotQA, 2WikiMultiHopQA) as well as the SynAgent-RAG test set. The central claim is that a small open-source model, after instruction tuning, achieves competitive performance with closed-source (GPT-4) and larger open-source (Llama-3-70B) agents while producing interpretable reasoning traces.

Significance. If the central claim holds, the framework would be a practical contribution: a trainable, open-source, small-backbone agent that unifies single-hop and multi-hop RAG and provides interpretable step-by-step traces. The release of the dataset and code is a strength for reproducibility, and the idea of distilling agent behavior via a multi-task conversational format is useful. However, the paper's headline comparison to GPT-4 and Llama-3-70B rests on an evaluation that is substantially self-referential, and the standard-benchmark comparison only uses GPT-3.5-based baselines. The significance of the contribution is therefore conditional on addressing the evaluation concerns outlined in the major comments. If those are resolved, the paper would be a solid empirical contribution to trainable open-source RAG agents.

major comments (4)

[§5.2, Table 3] The claim of being competitive with GPT-4 and Llama-3-70B is based entirely on Table 3, which is evaluated on the SynAgent-RAG test set. This test set is generated by the same GPT-4 teacher pipeline that produced the training set (Figures 3, 6-9), and the GPT-Score metric is computed by an LLM-based evaluator prompted with a GPT-4-designed rubric (Figure 11). The student is therefore evaluated on the teacher's own generated distribution and scored by a grader that is not shown to be independent of the teacher. Additionally, the comparison is performed under the oracle retrieval setting described in §5.1.2, where relevant documents are given directly, not on the realistic benchmark corpora used in Tables 1-2. Consequently, the abstract's claim of 'comparable performances with closed-source and larger open-source LLMs across various RAG benchmarks' is not tested in any independent setting; Tables 1-2 compare only against GPT-3.5-based baselines. I recommend adding experiments with GPT-4 and Llama-3-70B on the standard benchmarks, or at minimum validating GPT-Score against human judgments and reporting results on a test set drawn from a different distribution.
[§4.1.1, §5.1.2] No decontamination check is reported between the SynAgent-RAG training data (derived from Wikipedia Vital Articles Level 5) and the corpora and questions of the six evaluation benchmarks (SQuAD, Natural Questions, TriviaQA, MuSiQue, HotpotQA, 2WikiMultiHopQA). Since all benchmark corpora are Wikipedia-derived (Karpukhin et al. 2020, Trivedi et al. 2023), and Vital Articles include many popular entities and passages, there is a real possibility of training/test overlap. If overlap exists, the reported gains could reflect memorized content rather than the agent's learned search-and-reasoning behavior. The paper states the split is 'carefully divided' but gives no details on the split, and the Limitations section does not acknowledge this risk. I request a quantitative overlap analysis (e.g., n-gram overlap between training passages and benchmark passages/questions) and, if necessary, filtering of overlapping content.
[§5.1.1, Tables 1–2] All results are computed on 500 samples per dataset with a single run and no error bars, confidence intervals, or significance tests. The improvements over baselines in Tables 1 and 2 are generally a few points in F1 or EM, which could be within sampling noise. For example, in Table 2 on HotpotQA, Agent-UniRAG (No limit, 12/Yes) reports EM 50.2 vs. IRCoT's 45.8, but without variance we cannot assess whether this is meaningful. I recommend reporting bootstrap confidence intervals or multiple seeds, and running a paired significance test (e.g., bootstrap or approximate randomization) for the headline comparisons.
[Figure 11, §5.2] The GPT-Score metric is not validated. The prompt in Figure 11 instructs the grader to penalize 'missing or excess information compared to the reference answer.' Since the reference answers are generated by GPT-4 and the student is distilled from GPT-4, this metric is likely biased toward the student's output style and content. The paper provides no correlation with human judgments, no inter-annotator agreement, and no evidence that the LLM grader is consistent. At minimum, the paper should report the evaluator's agreement with human scores and ideally use a blind evaluation protocol.

minor comments (8)

[Table 1] The table formatting appears garbled: for example, the row 'Agent-UniRAG 1 8 / No 23.8 34.549.643.4 51.6 61.2 57.6 65.871.2' is not readable; the EM/F1/Acc values should be separated into columns (e.g., 23.8, 34.5, 49.6, etc.).
[Table 2] The row 'w/o Evidence Reflector' contains '57.94' which is likely a typo for '57.9' or '57.9 4'; the value should be aligned with the three column entries for 2WikiMultiHopQA.
[§4.1.1] The split of Vital Articles into training and testing is only described as 'carefully divided'; for reproducibility, please specify the number of articles, the split ratio, the random seed, and whether the split is at the article level or passage level.
[§5.1.2] The oracle retrieval condition used for Table 3 is a significant departure from the realistic retrieval used in Tables 1-2; this should be stated clearly in the main text and in the discussion of the GPT-4 comparison, not only in the setup description.
[Limitations] The Limitations section only mentions the computational cost of multiple LLM calls and the need to extend to non-RAG tasks; it does not mention the potential distributional overlap between training data and benchmarks or the self-referential nature of the GPT-4 evaluation. These should be acknowledged.
[Introduction and Related Work] The claim of being 'the first study to execute the unified RAG system in an end-to-end manner' is strong and would benefit from a more careful literature search and a softer phrasing.
[Figure 6 prompt] The prompt in Figure 6 has a duplicated item number ('3) The information...' appears twice); fix the numbering and ensure the prompt is complete.
[Figure 4] The question-type distribution is shown for the training set only; please also provide the distribution for the test set to confirm that the split is representative.

Circularity Check

1 steps flagged · score 4.0 of 10

GPT-4 parity claim rests on a GPT-4-generated test set and a GPT-4-style grader, but the core Agent-UniRAG framework is independently evaluated on six external benchmarks.

other [Section 4 (SynAgent-RAG construction) and Section 5.2, Table 3]
""SynAgent-RAG ... is achieved through a distillation approach (Semnani et al., 2023), where GPT-4 serves as the teacher model to generate data" (Sec. 4); "we also evaluate performance on the SynAgent-RAG test set" (Sec. 5.1.1); Table 3 caption: "Agent-UniRAG in compare with LLama-3-70B-Inst and GPT-4-Turbo on SynAgent-RAG test set"; Figure 11: "Prompt template for GPT4 to compare and score the predicted answer and the reference answer"."

The headline comparison to GPT-4 is the only place the paper directly compares Agent-UniRAG (Llama-3-8B) with GPT-4, and it is run on a test set produced by the same GPT-4 teacher pipeline that wrote the training questions, reference answers, and solution annotations. The evaluation metric GPT-Score is itself assigned by an LLM, with the appendix explicitly labeling the grader as GPT-4. Thus the claimed parity is measured on the teacher's own generated distribution and graded by a model of the same family, so the result reflects in-distribution distillation success more than independent parity evidence.

full rationale

The Agent-UniRAG architecture and training procedure are not formally circular: the agent loop, ReAct-style planning, evidence reflector, and memory are standard external ideas, and the six RAG benchmarks (SQuAD, NQ, TriviaQA, MuSiQue, HotpotQA, 2WikiMultiHopQA) provide an independent evaluation of the framework against prior methods. No self-citation chain or imported uniqueness theorem is load-bearing. The only substantive circularity is the self-referential nature of the SynAgent-RAG test set used for the GPT-4/Llama-3-70B parity claim: the teacher model generates the test questions and reference answers, the student is trained on the same teacher's solution traces, and the reported GPT-Score is produced by a GPT-4-based prompt. That makes the headline parity claim substantially in-distribution rather than an external validation. The paper also reports no decontamination check against the Wikipedia-derived external benchmarks; that is a correctness/leakage risk, not a circularity of derivation, so it is noted but does not further raise the circularity score.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the assumption that GPT-4-generated traces are a valid training signal, that the Wikipedia training data are disjoint from the benchmarks, and that GPT-4-based grading measures answer quality. Hyperparameters such as search budget and top-K are tuned by hand, and the best configuration is highlighted.

free parameters (5)

Maximum search steps (Max Search) = 1, 8, 12, or no limit
Chosen by hand as the agent's compute budget; the paper reports results for several values and highlights the best-performing configurations (12 or no limit).
Top-K retrieved documents = 8 or 12
Chosen by hand; larger K generally improves recall and final scores, so the headline results use K=12.
Dense reranker (BiEncoder E5) usage = enabled or disabled
A system choice that improves retrieval quality; the best results use it, but the contribution of the agent itself is then mixed with a stronger retriever.
Verification score threshold for SynAgent-RAG = 4 on a 0-5 GPT-4 score
Annotations with score below 4 are discarded from the dataset; this threshold is chosen by hand and changes the training distribution.
Training hyperparameters = lr=2e-5, batch=256, epochs=2
Standard instruction-tuning choices; they affect the distilled model's behavior but are not central to the agent design.

assumptions (5)

domain assumption GPT-4-generated synthetic annotations (thought, action, evidence) are high enough quality to distill agent reasoning into a small open-source LLM.
The whole training set is built by GPT-4 and filtered by GPT-4 scoring; no human evaluation or external validation is reported beyond the scoring threshold.
ad hoc to paper The split of Wikipedia Vital Articles into train/test prevents SynAgent-RAG from overlapping with the six benchmark corpora.
Benchmark corpora (SQuAD, NQ, TriviaQA, MuSiQue, HotpotQA, 2Wiki) are Wikipedia-derived, but no decontamination check is provided. If overlap exists, benchmark numbers are inflated.
domain assumption Hyperlinks between Wikipedia passages are reliable indicators of multi-hop evidence dependencies.
The dataset construction selects supporting passages based on GPT-4 scoring of hyperlinks; this heuristic may not generalize to other corpora.
domain assumption ReAct-style iterative loop is an appropriate universal controller for both single- and multi-hop RAG tasks.
The architecture assumes the LLM can learn when to stop searching and answer; the ablations support this, but only on multi-hop datasets and one configuration.
ad hoc to paper Evaluation with GPT-4 as an answer grader (GPT-Score) is a valid measure of answer quality, especially for comparing a GPT-4-distilled model against GPT-4.
GPT-Score is computed by the same model that generated the reference answers and served as teacher, potentially favoring the distilled student.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Agent-UniRAG: A Trainable Open-Source LLM Agent Framework for Unified Retrieval-Augmented Generation Systems." pith.science (2026). https://pith.science/paper/5GXVIBU2

@misc{pith2026250522571,
  author       = {Pith},
  title        = {Pith review of: Agent-UniRAG: A Trainable Open-Source LLM Agent Framework for Unified Retrieval-Augmented Generation Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5GXVIBU2}},
  note         = {Machine review of arXiv:2505.22571}
}

read the original abstract

This paper presents a novel approach for unified retrieval-augmented generation (RAG) systems using the recent emerging large language model (LLM) agent concept. Specifically, Agent LLM, which utilizes LLM as fundamental controllers, has become a promising approach to enable the interpretability of RAG tasks, especially for complex reasoning question-answering systems (e.g., multi-hop queries). Nonetheless, previous works mainly focus on solving RAG systems with either single-hop or multi-hop approaches separately, which limits the application of those approaches to real-world applications. In this study, we propose a trainable agent framework called Agent-UniRAG for unified retrieval-augmented LLM systems, which enhances the effectiveness and interpretability of RAG systems. The main idea is to design an LLM agent framework to solve RAG tasks step-by-step based on the complexity of the inputs, simultaneously including single-hop and multi-hop queries in an end-to-end manner. Furthermore, we introduce SynAgent-RAG, a synthetic dataset to enable the proposed agent framework for small open-source LLMs (e.g., Llama-3-8B). The results show comparable performances with closed-source and larger open-source LLMs across various RAG benchmarks. Our source code and dataset are publicly available for further exploitation.

Figures

Figures reproduced from arXiv: 2505.22571 by the authors.

**Figure 2.** Overall design of Agent-UniRAG. 3.1 Planning Module Leveraging the reasoning capabilities of modern LLMs, this module is designed to systematically determine the necessary actions required to address a user’s request (Input Query) at each step of the process. Specifically, the agent decides between two primary actions at each decision point: • Action: Search – This action is triggered when the agent needs to acquire… view at source ↗

**Figure 3.** Overview of the proposed SynAgent-RAG Dataset [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (11 more)

**Figure 4.** Figure 4: Question type distribution on the training set. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: Prompt template for GPT4 to find related section content from articles. [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]

**Figure 6.** Figure 6: Prompt template for GPT4 to generate multi-hop questions. [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]

**Figure 7.** Figure 7: Prompt template for GPT4 to generate single-hop questions. [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

**Figure 8.** Figure 8: Prompt template for GPT4 to extract evidence from a list of sources [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]

**Figure 9.** Figure 9: Prompt template for GPT4 to reason, use tools and provide the final answer for a question [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]

**Figure 10.** Figure 10: Prompt template to extract short answer from long answer [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]

**Figure 11.** Figure 11: Prompt template for GPT4 to compare and score the predicted answer and the reference answer. [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]

**Figure 12.** Figure 12: Training prompt template for Agent-UniRAG to reason and use tools. Loss is computed only on the red [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]

**Figure 13.** Figure 13: Training prompt template for Agent-UniRAG extract related evidence for a query from sources of [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]

**Figure 14.** Figure 14: Training prompt template for Agent-UniRAG to provide final answer from gathered evidence. Loss is [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 34 canonical work pages

[1]

You can select ONLY one section title per article

work page
[2]

rationale

Response must be in the following JSON format: [ { "rationale": "condensed and short reason why you select the section", "article_title": "article_title", "section_title": "article_section_title" },... ]

work page
[3]

### Clarification:

You can select up to {{k}} most proper section titles Figure 5: Prompt template for GPT4 to find related section content from articles. ### Clarification:

work page
[4]

article_title

Do not mention the source of information in the question or the answer. Figure 6: Prompt template for GPT4 to generate multi-hop questions. ### You will be given a source of article. Your job is to create a relevant question to the source and then provide the answer for the question based on the provided source: ### Source: { "article_title": "actual arti...

work page
[5]

question

Do NOT mention the source in the question or in the answer. ### Response MUST be in the following JSON format: { "question": "Your question here", "answer": "the detailed answer to the question" } Figure 7: Prompt template for GPT4 to generate single-hop questions. ### Task: Synthesize a condensed text evidence from given sources to support a search query...

work page
[8]

A multi-hop question is a question that requires multiple inferential leaps or accessing several pieces of information from different sources to arrive at a final answer

work page
[9]

article_title

You will be given sources of articles, your job is to generate a multi-hop question and then provide the answer for the question based on the provided sources. ### Sources [ { "article_title": "actual article title", "section_title": "actual section title", "content": "actual content" },... ] ### Notes:

work page
[10]

The question cannot be answered by relying on any single article alone but instead requires the solver to gradually gather and search for pieces of evidence within ALL the provided sources then understand and link information to take the next action, and finally give back the answer

work page

Show all 37 references

[11]

Make sure the question flows logically and is unambiguous
[13]

question

Response in the following JSON format: {"question": "your question", "answer": "correct answer for the question"}
[15]

The question must related to the content of the source
[16]

Make sure the question is simple enough and unambiguous
[17]

The question requires synthesizing information from the source to answer
[18]

The information in the answer MUST be derived from the sources
[20]

Clarity: Evidence must be clear, concise
[21]

Conciseness: Evidence must be presented in a succinct manner, condensed and AVOIDING unnecessary details
[22]

Relevance: Evidence must directly correspond and relevant to the search query
[23]

Source Integrity: Only use information from the provided sources, AVOIDING generated or unnecessary information
[24]

evidence

If multiple part of a source is relevant to the search query, combine them into one element in the response list. ### Response MUST be in a JSON list as below: [ { "evidence": "condensed text supporting the search query from a source", "source_id": "an identifier of the source...
[25]

Reasoning step by step how you will use the tool to solve the question
[26]

You can only use one tool each time then you get the response and continue
[27]

Provided that you DO NOT have any initial knowledge about the information mentioned in the question and DO NOT generate facts or evidence yourself
[28]

Provide a CLEAR and CONCISE answer
[29]

(one sentence) ### Search Input: Format the search query input for the search_engine tool as a JSON object, correctly representing input parameters

Format responses to utilize the search_engine tool as follows: ### Thought: A short and condensed rationale for using the search_engine tool. (one sentence) ### Search Input: Format the search query input for the search_engine tool as a JSON object, correctly representing inpu...
[30]

short_extracted_answers

Your response MUST be in JSON format {"short_extracted_answers": ["extracted answer 1", ...]}
[31]

The extracted answers MUST be united and interchangeable, try to combine nearby words in the Reference Answer to from an answer
[32]

Do not generate answers or information yourself
[33]

If the question is a yes/no question, then you should base it on the reference answer to return yes or no as the extracted answer
[34]

short_extracted_answers

If you can not extract the answer or the answer is not provided in the Reference Answer, then respond: {"short_extracted_answers": null} ### Examples: Question: What percentage of French publishing houses were in Paris in the 1970s? Reference Answer: In the 1970s, 80 percent o...

1917
[35]

The score MUST be an integer range from 0 to 5
[36]

The content of the predicted answer should be relevant and focus on the question
[37]

Any missing or excess information in the predicted answer compared to the reference answer will be penalized in the final score
[38]

rationale

If the question is a question that requires the analysis of information, then you should reinforce the above criteria. ### Question: {{question}} ### Reference Answer: {{reference answer}} ### Predicted Answer: {{predicted answer}} ### Note: Your response MUST be in the follow...
[2020]

CoRR, abs/2004.04906

Dense passage retrieval for open-domain ques- tion answering. CoRR, abs/2004.04906. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Lli...

2004 arXiv
[2022]

In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 2206–2240

Improving language models by retrieving from trillions of tokens. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 2206–2240. PMLR. Wenqi Fan, Yujuan Ding, Liang...

2022 arXiv
[2023]

article_title

Hugginggpt: Solving AI tasks with chatgpt and its friends in hugging face. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023. Noah Shinn, Federico...

2023 arXiv
[2413]

Yongliang Shen, Kaitao Song, Xu Tan, Dong- sheng Li, Weiming Lu, and Yueting Zhuang

Association for Computational Linguistics. Yongliang Shen, Kaitao Song, Xu Tan, Dong- sheng Li, Weiming Lu, and Yueting Zhuang

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

You can select ONLY one section title per article

work page

[2] [2]

rationale

Response must be in the following JSON format: [ { "rationale": "condensed and short reason why you select the section", "article_title": "article_title", "section_title": "article_section_title" },... ]

work page

[3] [3]

### Clarification:

You can select up to {{k}} most proper section titles Figure 5: Prompt template for GPT4 to find related section content from articles. ### Clarification:

work page

[4] [4]

article_title

Do not mention the source of information in the question or the answer. Figure 6: Prompt template for GPT4 to generate multi-hop questions. ### You will be given a source of article. Your job is to create a relevant question to the source and then provide the answer for the question based on the provided source: ### Source: { "article_title": "actual arti...

work page

[5] [5]

question

Do NOT mention the source in the question or in the answer. ### Response MUST be in the following JSON format: { "question": "Your question here", "answer": "the detailed answer to the question" } Figure 7: Prompt template for GPT4 to generate single-hop questions. ### Task: Synthesize a condensed text evidence from given sources to support a search query...

work page

[6] [8]

A multi-hop question is a question that requires multiple inferential leaps or accessing several pieces of information from different sources to arrive at a final answer

work page

[7] [9]

article_title

You will be given sources of articles, your job is to generate a multi-hop question and then provide the answer for the question based on the provided sources. ### Sources [ { "article_title": "actual article title", "section_title": "actual section title", "content": "actual content" },... ] ### Notes:

work page

[8] [10]

The question cannot be answered by relying on any single article alone but instead requires the solver to gradually gather and search for pieces of evidence within ALL the provided sources then understand and link information to take the next action, and finally give back the answer

work page