Pith. sign in

REVIEW 5 major objections 7 minor 1 cited by

SRSA: A Cost-Efficient Strategy-Router Search Agent for Real-world Human-Machine Interactions

T0 review · 5 major / 7 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A strategy router that matches each user query to one of three search plans—rewrite, parallel, or stepwise—can produce more informative and complete answers than a single search or a naive ReAct loop, without fine-tuning the LLM.

desk verdict Useful new dataset and a plausible cost-aware router, but the evaluation is too weak to support the headline quality and cost claims. read the letter →

arxiv 2411.14574 v1 pith:QYRSO727 submitted 2024-11-21 cs.AI

Yaqi Wang , Haipei Xu This is my paper · ORCID
classification cs.AI
keywords strategyroutersearchagentretrieval-augmentedgenerationlargelanguagemodelsqueryrewritingReActcontextualqueriesLLMevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Real chatbot queries are often long, personal, and tangled: they mix several implicit sub-questions and depend on context a search engine cannot see. The paper proposes a Strategy-Router Search Agent (SRSA) that first classifies each query into one of three search strategies—rewrite-and-search, parallel sub-question search, or stepwise planning with per-step filtering—and then feeds the resulting reference passages into a frozen LLM to write the final answer. The central claim is that this routing fixes a degeneration seen in earlier iterative agents: a ReAct-style agent that just thinks-acts-observes repeatedly performs worse than a single-round search on these contextual queries, while SRSA's planning strategy compresses and filters each search round and outperforms both baselines on informativeness and completeness. If true, it means better answers for realistic human-chatbot interactions can be obtained without fine-tuning, and at lower cost because simple queries are sent to cheap strategies.

What carries the argument

The central mechanism is the strategy-router module and the three parameter-free search strategies it selects among. The router is a single LLM call that, given the user query (with a time module), outputs both a strategy label $\mathcal{S} \in \{D, P, R\}$ and a set of search suggestions $\mathrm{sug}(\mathcal{S})$. Direct search ($D$) rewrites the query and searches once; parallel search ($P$) generates several sub-questions, searches them concurrently, and concatenates the results; planning search ($R$) runs a loop that compresses and filters the previous round's results, evaluates whether they are relevant, rewrites the query if they are not, and stops when the accumulated memory suffices. The final answer is generated by a RAG prompt from the original query plus the strategy's curated reference passage. This design is what lets the system balance cost and quality: easy queries go to $D$, multi-concept queries to $P$, and sequential, context-dependent queries to $R$.

What would settle it

Run the same CQED comparison with human raters (or a second independent LLM judge) scoring the answers blind to agent identity, and check whether SRSA's advantage over the simple search agent on informativeness and completeness persists. If it shrinks or disappears when answer length and structure are controlled for, the reported gains are an artifact of the judge rather than of the routing.

Watch

Extended reading notes

Core claim

On its own terms, the paper's core discovery is that a strategy router—a single LLM prompt that outputs a strategy label plus search suggestions—can allocate queries among three search strategies so that the system as a whole is both more informative and more complete than either a single-search agent or a ReAct-based agent, while spending less compute on easy queries. The experimental result on the Mistral model shows SRSA significantly beats the ReAct agent and the simple search agent on informativeness (t=14.98, p<0.01) and completeness (t=4.88, p<0.01), with no significant loss on novelty or actionability. The paper also demonstrates that a naive ReAct agent "degenerates" on contextual queries, underperforming even the simple search agent, and attributes this to irrelevant search results accumulating in the iterative history. The router's planning strategy addresses this by summarizing and filtering each iteration's results before deciding the next search. In addition, the paper contributes the Contextual Query Enhancement Dataset (CQED), built from Reddit threads, to test such contextual queries.

Load-bearing premise

The head-to-head comparisons assume the LLM judge's scores for informativeness, completeness, novelty, and actionability are unbiased and meaningful; the paper reports only a vague manual check of 8 data sets, with no inter-rater agreement or evidence that the judge is not favoring longer, more structured answers.

Editorial extensions

If this is right

  • A straightforward ReAct-style search loop can be worse than a single search on context-rich queries, so search agents should filter or compress intermediate results rather than accumulate all of them.
  • Query rephrasing alone, before a single search, improves informativeness and completeness on lengthy queries, even for models that cannot follow complex routing prompts.
  • A frozen LLM with prompt-based routing can match or beat fine-tuned search agents on contextual queries, which lowers the cost of deployment.
  • The CQED dataset provides a way to measure whether search agents understand user context, complementing existing QA datasets that assume clean factual questions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The routing principle likely extends beyond search to other tool-using agents: instead of always running the same expensive loop, an agent could classify a task's complexity first and dispatch to a cheaper or more elaborate tool sequence accordingly.
  • A testable extension is to replace the LLM judge with human raters or a different judge model; if the SRSA advantage shrinks, the observed gains may be partly an artifact of judge preferences for longer, structured answers.
  • The degeneration of ReAct on contextual queries suggests a broader caution about iterative reasoning agents: without filtering, errors and irrelevance compound over turns, which may also affect multi-step planning agents in other domains.
  • The router's cost savings could be quantified more precisely by measuring token usage or response latency per strategy, not just quality scores.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper proposes SRSA, a strategy-router search agent that classifies each contextual query into one of three search strategies (direct, parallel, planning) and then generates the final answer from the retrieved supporting documents. The authors introduce a new dataset, CQED, of realistic, context-rich user queries, and evaluate SRSA against a single-round simple search agent and a ReAct-based agent using GPT-4o-mini as an LLM judge on informativeness, completeness, novelty, and actionability. The reported results, based mainly on Mistral-7B, claim significant gains in informativeness and completeness over both baselines, and a cost-efficiency advantage from routing easy queries to cheaper strategies.

Significance. If the empirical claims were properly supported, the idea of routing queries to different search strategies to balance quality and cost would be practically valuable for real-world chatbot deployments, and the CQED dataset would fill a genuine gap in evaluation resources. The paper also contributes a clear description of a planning search strategy with a compression/filtering step aimed at avoiding the degeneration seen with ReAct. However, the current evidence is not sufficient to establish these contributions because the evaluation rests on an unvalidated LLM judge, the cost claim is never measured, and the router's effectiveness is inferred from a null result rather than a direct ablation.

major comments (5)
  1. [§IV-B.3, §IV-C] All comparative conclusions—the t-tests in Section IV-C.3 (t=14.98, p<0.01 for informativeness), the ReAct degeneration finding in Table I, and the strategy comparisons in Table II—depend entirely on quality scores assigned by a single LLM judge, GPT-4o-mini, whose validity is never established. The paper reports only 'we manually evaluated 8 data sets' with no details on the number of items, the evaluation procedure, inter-annotator agreement, or correlation with the LLM judge's scores. Because the judge sees all three outputs simultaneously and the rubrics for informativeness and completeness reward longer, more structured answers, the reported superiority of SRSA may be an artifact of answer format rather than genuine quality. The authors should provide a human-annotation study with agreement statistics, plus a length- or position-bias analysis, to validate the judge before the headline quality claims can be accepted.
  2. [§IV-C.4, Table II] The claim that the strategy router is effective is supported only by a null result: after routing, there are few significant differences between the strategies. The paper states in the same section that the authors 'aim to demonstrate that without the search router, when all questions are processed through a single strategy, there are significant differences in the quality of query answers,' but this no-router comparison is never reported. Without this comparison, the observed null result is equally consistent with the router being unnecessary, because all strategies may perform similarly regardless of routing. The authors should add a direct ablation comparing quality when all queries are forced through each strategy against the router-assigned strategy, and ideally also measure router accuracy against ground-truth labels.
  3. [Abstract, Title, §IV-C.4, §V] The paper is titled and framed as 'cost-efficient' and claims to balance quality and computational cost, yet no cost measurements are reported anywhere. There are no counts of LLM calls, token usage, inference time, or API costs for SRSA versus the baselines. The router's cost-saving claim is only supported by the qualitative statement that simple queries are routed to the cheaper direct strategy. The authors should report quantitative cost metrics (e.g., average LLM calls per query, total tokens, wall-clock time) for each agent across the CQED dataset.
  4. [§IV-C, §V (Limitation)] The successful full implementation of SRSA is demonstrated only with Mistral-7B-Instruct-v0.3; the paper acknowledges that Gemma and Llama were unable to follow the required formatted output and therefore defaulted to the direct strategy. While this is noted as a limitation, the abstract and conclusion make general claims that SRSA 'provides an approach' that 'effectively and efficiently parses complex user queries.' Given that three of the core modules (router, parallel, planning) rely heavily on formatted instruction following, the paper should either restrict its claims to models with strong instruction-following capabilities or include results from at least one additional model that can execute the full workflow.
  5. [§IV-B.3] The statistical analysis assumes that the automatic evaluation scores follow a truncated normal distribution, but no justification is given for this assumption, and the scores are discrete integers on a 0–5 scale. Additionally, the paper performs multiple pairwise t-tests across four metrics and three strategy comparisons without any correction for multiple comparisons, which inflates the risk of false positives. The authors should justify the t-test assumption or use a non-parametric test, and report adjusted p-values or a clear multiple-comparison strategy.
minor comments (7)
  1. [§IV-C.3] The sentence reporting t-statistics of 14.9827 and 4.8846 does not specify which comparisons these correspond to (SRSA vs. simple search, or SRSA vs. ReAct, and for which metric each). The text should be explicit.
  2. [Table II caption] The caption says 'After searching the router'; this should be 'After routing by the router' or 'After routing.'
  3. [§IV-B.3] The phrase 'we manually evaluated 8 data sets' is ambiguous and likely means '8 data points' or '8 examples.' The number of manually evaluated items and the selection procedure should be stated clearly.
  4. [§IV-A.2] The dataset construction paragraph says 'a total of 182 answers,' but it should clarify whether this is 182 queries, 182 reference answers, or both. Also, the statement that warm-up queries are removed should report the final dataset size.
  5. [§II-C.1] Reference [21] is cited for Interleaving Retrieval with Chain-of-Thought (IR-CoT), but [21] is Query2doc; the IR-CoT reference appears later as [10]. The citation numbering should be corrected.
  6. [§III-B] The notation is confusing because the letter S is used both for the chosen strategy (S, sug(S)) and for the search function S(·). Using two different symbols would improve readability.
  7. [§IV-C.4] The bar charts in Figure 7 are not described with exact numbers in the text; reporting the mean scores for each strategy would make the figure self-contained and easier to verify.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's central claims rest on empirical LLM-judge comparisons, not on self-referential derivations or fitted parameters.

full rationale

SRSA's derivation chain is empirical rather than definitional. The router, the three search strategies, and the final answer generator are specified as prompted LLM functions (Section III), and their outputs are compared with two baselines through a shared Tavily search API and a single GPT-4o-mini judge (Sections IV-B and IV-C). No parameter is fitted to the evaluation outcomes, and no central claim is defined in terms of its own conclusion. The router validation in Section IV-C.4 is admittedly weak: Table II's absence of significant score differences is an underdetermined basis for 'appropriately assigning queries,' and it would also be consistent with an unnecessary router; however, this is an interpretive and generalization problem, not a reduction of the conclusion to the premise. Similarly, the reliance on an unvalidated LLM judge, the vague manual check of '8 data sets,' and the self-created CQED dataset raise validity and external-benchmark concerns, but they do not make the reported comparisons true by construction. There are no load-bearing self-citations or imported uniqueness theorems, and the paper's limitation section openly acknowledges the single-model generalizability issue. Hence no circular step can be exhibited.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central comparative claims depend on the validity of the retrieval API, the LLM judge, the new dataset, the router's decisions, and an unverified statistical assumption. None of these are independently validated in the paper, which raises the evidential burden on the headline comparison.

assumptions (5)
  • domain assumption Tavily Search API returns relevant, reliable, and sufficiently comprehensive web results for all queries.
    Used as the sole retrieval tool; no quality checks on the retrieved documents are reported (Section IV.B.2).
  • domain assumption GPT-4o-mini scores answers validly on the four quality dimensions without systematic bias.
    The validation is only a vague manual check of '8 data sets' with no inter-rater agreement (Section IV.B.3).
  • domain assumption CQED queries are representative of authentic human-chatbot interactions.
    The dataset is constructed with LLM assistance and manual selection from Reddit posts, with no external validation or comparison to real chatbot logs (Section IV.A.2).
  • domain assumption The LLM's strategy classification is accurate and beneficial for all queries.
    Router accuracy is only inferred indirectly from the absence of quality differences between strategies; no human labels or oracle routing are used (Section IV.C.4).
  • ad hoc to paper The t-test with an assumed truncated normal distribution is valid for the evaluation scores.
    The distributional assumption is stated without empirical justification and affects all reported p-values (Section IV.B.3).

how reviews work

0 comments
Cite this review

Pith. "Pith review of SRSA: A Cost-Efficient Strategy-Router Search Agent for Real-world Human-Machine Interactions." pith.science (2026). https://pith.science/paper/QYRSO727

@misc{pith2026241114574,
  author       = {Pith},
  title        = {Pith review of: SRSA: A Cost-Efficient Strategy-Router Search Agent for Real-world Human-Machine Interactions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QYRSO727}},
  note         = {Machine review of arXiv:2411.14574}
}
read the original abstract

Recently, as Large Language Models (LLMs) have shown impressive emerging capabilities and gained widespread popularity, research on LLM-based search agents has proliferated. In real-world situations, users often input contextual and highly personalized queries to chatbots, challenging LLMs to capture context and generate appropriate answers. However, much of the prior research has not focused specifically on authentic human-machine dialogue scenarios. It also ignores the important balance between response quality and computational cost by forcing all queries to follow the same agent process. To address these gaps, we propose a Strategy-Router Search Agent (SRSA), routing different queries to appropriate search strategies and enabling fine-grained serial searches to obtain high-quality results at a relatively low cost. To evaluate our work, we introduce a new dataset, Contextual Query Enhancement Dataset (CQED), comprising contextual queries to simulate authentic and daily interactions between humans and chatbots. Using LLM-based automatic evaluation metrics, we assessed SRSA's performance in terms of informativeness, completeness, novelty, and actionability. To conclude, SRSA provides an approach that resolves the issue of simple serial searches leading to degenerate answers for lengthy and contextual queries, effectively and efficiently parses complex user queries, and generates more comprehensive and informative responses without fine-tuning an LLM.

Figures

Figures reproduced from arXiv: 2411.14574 by the authors.

Figure 1
Figure 1. Strategy-Router Search Agent (SRSA) Workflow [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Workflow of planning search strategy module [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Comparison of the CQED and other QA datasets [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Construction Process of CQED The maximum iteration times are set to be 5. Generally, after receiving a query, the ReAct-based search agent analyzes and thinks about what to search for (Thought step), then searches (using the Search Engine API, discussed in the next sec…
Figure 5
Figure 5. Figure 5: Compare Llama and gemma’s performance on SRSA(Direct) and simple search agent, indicating that rephrasing before [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Wining Proportions of different agents. The naive ReAct-based agent has the lowest winning rate in all metrics, lower [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Comparison SRSA to baseline models and different search strategies’ performance [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Agentic Systems: A Guide to Transforming Industries with Vertical AI Agents

    cs.MA 2025-01 conditional novelty 3.0 of 10

    A practitioner-oriented survey proposing a 'Cognitive Skills' module as a standardized building block for vertical AI agents.

Reference graph

Works this paper leans on

44 extracted references · 19 canonical work pages · cited by 1 Pith paper

  1. [1]

    The rise and potential of large language model based agents: A survey,

    Z. Xi et al., “The rise and potential of large language model based agents: A survey,” arXiv preprint arXiv:2309.07864, 2023

    arXiv 2023

  2. [2]

    Siren’s song in the AI ocean: a survey on hallucination in large language models,

    Y . Zhang et al., “Siren’s song in the AI ocean: a survey on hallucination in large language models,” arXiv preprint arXiv:2309.01219, 2023

    arXiv 2023

  3. [3]

    Metacognitive retrieval- augmented large language models,

    Y . Zhou, Z. Liu, J. Jin, J. Y . Nie, and Z. Dou, “Metacognitive retrieval- augmented large language models,” in Proc. ACM Web Conf. 2024, 2024, pp. 1453–1463

    work page 2024

  4. [4]

    RA-ISF: Learning to Answer and Understand from Retrieval Augmentation via Iterative Self-Feedback,

    Y . Liu et al., “RA-ISF: Learning to Answer and Understand from Retrieval Augmentation via Iterative Self-Feedback,” arXiv preprint arXiv:2403.06840, 2024

    arXiv 2024

  5. [5]

    ERAGent: Enhancing Retrieval-Augmented Language Models with Improved Accuracy, Efficiency, and Personalization,

    Y . Shi et al., “ERAGent: Enhancing Retrieval-Augmented Language Models with Improved Accuracy, Efficiency, and Personalization,” arXiv preprint arXiv:2405.06683, 2024

    arXiv 2024

  6. [6]

    BIDER: Bridging Knowledge Inconsistency for Efficient Retrieval-Augmented LLMs via Key Sup- porting Evidence,

    J. Jin, Y . Zhu, Y . Zhou, and Z. Dou, “BIDER: Bridging Knowledge Inconsistency for Efficient Retrieval-Augmented LLMs via Key Sup- porting Evidence,” arXiv preprint arXiv:2402.12174, 2024

    arXiv 2024

  7. [7]

    React: Synergizing reasoning and acting in language models,

    S. Yao et al., “React: Synergizing reasoning and acting in language models,” arXiv preprint arXiv:2210.03629, 2022

    arXiv 2022

  8. [8]

    Large language models can be easily distracted by irrelevant context,

    F. Shi et al., “Large language models can be easily distracted by irrelevant context,” in Proc. Int. Conf. Mach. Learn., PMLR, 2023, pp. 31210–31227

    work page 2023

Show all 44 references
  1. [9]

    Large language models know your contextual search intent: A prompting framework for conversational search,

    K. Mao, Z. Dou, F. Mo, J. Hou, H. Chen, and H. Qian, “Large language models know your contextual search intent: A prompting framework for conversational search,” arXiv preprint arXiv:2303.06573, 2023

    2023 arXiv

  2. [10]

    Interleav- ing retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions,

    H. Trivedi, N. Balasubramanian, T. Khot, and A. Sabharwal, “Interleav- ing retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions,” arXiv preprint arXiv:2212.10509, 2022

    2022 arXiv

  3. [11]

    A systematic survey of prompt engineering in large language models: Techniques and applications,

    P. Sahoo, A. K. Singh, S. Saha, V . Jain, S. Mondal, and A. Chadha, “A systematic survey of prompt engineering in large language models: Techniques and applications,” arXiv preprint arXiv:2402.07927 , 2024. Comparison Metric w/ Search Router t-stat p-value Direct vs. Parallel ...

    2024 arXiv

  4. [12]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei et al., “Chain-of-thought prompting elicits reasoning in large language models,” Advances in Neural Information Processing Systems, vol. 35, pp. 24824–24837, 2022

    2022

  5. [13]

    A survey on in-context learning,

    Q. Dong et al., “A survey on in-context learning,” arXiv:2301.00234, 2022

    2022 arXiv

  6. [14]

    Retrieval-augmented generation for knowledge-intensive NLP tasks,

    P. Lewis et al., “Retrieval-augmented generation for knowledge-intensive NLP tasks,” Advances in Neural Information Processing Systems, vol. 33, pp. 9459–9474, 2020

    2020

  7. [15]

    MetaGPT: Meta Programming for a Multi-Agent Collaborative Framework,

    S. Hong et al., “MetaGPT: Meta Programming for a Multi-Agent Collaborative Framework,” arXiv:2308.00352, 2023

    2023 arXiv

  8. [16]

    Data Interpreter: a LLM Agent For Data Science,

    S. Hong et al., “Data Interpreter: a LLM Agent For Data Science,” arXiv:2402.18679, 2024

    2024 arXiv

  9. [17]

    Introducing Devin: The First AI Software Engineer,

    Cognition Labs, “Introducing Devin: The First AI Software Engineer,”

  10. [18]

    Toolllm: Facilitating large language models to master 16000+ real-world apis,

    Y . Qin et al., “Toolllm: Facilitating large language models to master 16000+ real-world apis,” arXiv:2307.16789, 2023

    2023 arXiv

  11. [19]

    Webgpt: Browser-assisted question-answering with human feedback,

    R. Nakano et al., “Webgpt: Browser-assisted question-answering with human feedback,” arXiv preprint arXiv:2112.09332 , 2021

    2021 arXiv

  12. [20]

    Query rewrit- ing for retrieval-augmented large language models,

    X. Ma, Y . Gong, P. He, H. Zhao, and N. Duan, “Query rewrit- ing for retrieval-augmented large language models,” arXiv preprint arXiv:2305.14283, 2023

    2023 arXiv

  13. [21]

    Query2doc: Query expansion with large language models,

    L. Wang, N. Yang, and F. Wei, “Query2doc: Query expansion with large language models,” arXiv preprint arXiv:2303.07678 , 2023

    2023 arXiv

  14. [22]

    Knowledge refinement via interaction between search engines and large language models,

    J. Feng, C. Tao, X. Geng, T. Shen, C. Xu, G. Long, D. Zhao, and D. Jiang, “Knowledge refinement via interaction between search engines and large language models,” arXiv preprint arXiv:2305.07402 , 2023

    2023 arXiv

  15. [23]

    Know where to go: Make LLM a relevant, responsible, and trustworthy searcher,

    X. Shi, J. Liu, Y . Liu, Q. Cheng, and W. Lu, “Know where to go: Make LLM a relevant, responsible, and trustworthy searcher,” arXiv preprint arXiv:2310.12443, 2023

    2023 arXiv

  16. [24]

    Paraphrasing adaptation for web search ranking,

    C. Wang, N. Duan, M. Zhou, and M. Zhang, “Paraphrasing adaptation for web search ranking,” in Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (V olume 2: Short Papers) , 2013, pp. 41-46

    2013

  17. [25]

    Query expansion by prompting large language models,

    R. Jagerman, H. Zhuang, Z. Qin, X. Wang, and M. Bendersky, “Query expansion by prompting large language models,” arXiv preprint arXiv:2305.03653, 2023

    2023 arXiv

  18. [26]

    Least-to-most prompting enables complex reasoning in large language models,

    D. Zhou et al., “Least-to-most prompting enables complex reasoning in large language models,” arXiv:2205.10625, 2022

    2022 arXiv

  19. [27]

    When not to trust language models: Investigating effectiveness and limitations of parametric and non-parametric memories,

    A. Mallen, A. Asai, V . Zhong, R. Das, H. Hajishirzi, and D. Khashabi, “When not to trust language models: Investigating effectiveness and limitations of parametric and non-parametric memories,” arXiv preprint, 2022

    2022

  20. [28]

    TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension,

    M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer, “TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension,” arXiv e-prints , pp. arXiv:1705.03551, 2017

    2017 arXiv

  21. [29]

    HotpotQA: A dataset for diverse, explainable multi-hop question answering,

    Z. Yang et al., “HotpotQA: A dataset for diverse, explainable multi-hop question answering,” arXiv:1809.09600, 2018

    2018 arXiv

  22. [30]

    Think you have solved question answering? Try ARC, the AI2 reasoning challenge,

    P. Clark et al., “Think you have solved question answering? Try ARC, the AI2 reasoning challenge,” arXiv:1803.05457v1, 2018

    2018 arXiv

  23. [31]

    Python Reddit API Wrapper,

    PRAW, “Python Reddit API Wrapper,” GitHub repository, 2024. [On- line]. Available: https://github.com/praw-dev/praw

    2024

  24. [32]

    Gemma 2 model

    Google, “Gemma 2 model” Hugging Face, 2024. [Online]. Available: https://huggingface.co/google/gemma-2-2b-it

    2024

  25. [33]

    Llama 3 Model,

    AI@Meta, “Llama 3 Model,” GitHub Repository, 2024. [Online]. Avail- able: https://github.com/meta-llama/llama3/tree/main

    2024

  26. [34]

    Model Card for Mistral-7B-Instruct-v0.3,

    Mistral AI Team, “Model Card for Mistral-7B-Instruct-v0.3,” Hugging Face, 2024. [Online]. Available: https://huggingface.co/mistralai/Mistral- 7B-Instruct-v0.3

    2024

  27. [35]

    Tavily Search API,

    Tavily AI, “Tavily Search API,” GitHub Repository, 2024. [Online]. Available: https://github.com/tavily-ai/tavily-python

    2024

  28. [36]

    SerpAPI: Real-time search engine results API,

    SerpAPI, “SerpAPI: Real-time search engine results API,” SerpAPI,

  29. [37]

    Custom Search JSON API,

    Google, “Custom Search JSON API,” Google Developers,

  30. [38]

    Available: https://serpapi.com/

    [Online]. Available: https://serpapi.com/

  31. [39]

    A closer look into automatic evaluation using large language models,

    C.-H. Chiang and H.-Y . Lee, “A closer look into automatic evaluation using large language models,” arXiv:2310.05657, 2023

    2023 arXiv

  32. [40]

    Available: https://developers.google.com/custom- search/v1/overview

    [Online]. Available: https://developers.google.com/custom- search/v1/overview

  33. [41]

    Tavily API Documentation,

    Tavily, “Tavily API Documentation,” Tavily Documentation, 2024. [On- line]. Available: https://docs.tavily.com/docs/welcome

    2024

  34. [43]

    Can large language models be an alternative to human evaluations?,

    C.-H. Chiang and H.-Y . Lee, “Can large language models be an alternative to human evaluations?,” arXiv:2305.01937, 2023

    2023 arXiv

  35. [44]

    Lost in the middle: How language models use long contexts,

    N. F. Liu et al., “Lost in the middle: How language models use long contexts,” Trans. Assoc. Comput. Linguistics, vol. 12, pp. 157-173, 2024

    2024

  36. [2024]

    Available: https://www.cognition-labs.com/introducing- devin

    [Online]. Available: https://www.cognition-labs.com/introducing- devin

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.