Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Benchmarking Deep Search over Heterogeneous Enterprise Data

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces HERB, a benchmark showing that state-of-the-art RAG and agentic systems score only 32.96 on enterprise deep-search questions because retrieval fails to gather evidence scattered across Slack, meetings, documents, and…

desk verdict A genuinely useful, well-built heterogeneous RAG benchmark whose absolute scores are uncalibrated until a human baseline or judge validation shows up. read the letter →

arxiv 2506.23139 v1 pith:EOIDI3MU submitted 2025-06-29 cs.CL cs.AI

Prafulla Kumar Choubey , Xiangyu Peng , Shilpa Bhagavath , Kung-Hsiang Huang , Caiming Xiong , Chien-Sheng Wu This is my paper
classification cs.CLcs.AI
keywords deepsearchretrieval-augmentedgenerationenterpriseRAGbenchmarkmulti-hopreasoningheterogeneousdataagenticunanswerablequeriesretrievalbottleneck
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces HERB, a benchmark for deep search in enterprises—a retrieval-centric task in which answering a question means deciding what to look for and where, across sources as different as Slack messages, meeting transcripts, internal documents, GitHub pull requests, URLs, and employee and customer records. It argues that existing multi-hop RAG benchmarks are artificial and shallow, and it builds HERB with a query-first pipeline: human experts define realistic search intents and reasoning paths, then LLMs simulate product planning, development, and support workflows that generate the evidence needed to answer them. The benchmark contains 815 answerable and 699 unanswerable queries over 39,190 heterogeneous artifacts, with event timestamps, duplicate names, and hand-designed distractors adding realistic noise. Its headline result is that the best evaluated agentic system, ReAct with GPT-4o, averages only 32.96, and the paper attributes this to retrieval being the main bottleneck: systems retrieve only part of the necessary evidence and then reason over incomplete context. If HERB captures real enterprise difficulty, current RAG systems are far from solving deep-search tasks, and progress depends on retrieving all needed evidence across heterogeneous sources rather than on stronger reasoning alone.

What carries the argument

The load-bearing mechanism is the workflow-guided, query-first synthesis pipeline. Rather than clustering documents and generating questions post hoc, HERB starts from 41 manually defined query templates and paired reasoning scenarios, then simulates nine enterprise workflows (three each for planning, development, and support) in which virtual teams act out the activities that produce the evidence. Output artifacts—Slack threads, meeting transcripts and chats, documents, GitHub pull requests, URLs, employee and customer metadata—are timestamped, linked to the queries they support, and seeded with distractors such as product renames, overlapping discussions, cross-product PRs, and legacy feedback mentions. Answer construction is deterministic and query-specific: the pipeline aggregates linked artifacts, extracts speakers and authors, resolves names to employee IDs, and applies the human-designed reasoning step, avoiding LLM-generated answers. The evaluation harness then isolates the retrieval contribution by comparing full-RAG, product-specific long-context, product-specific RAG, and oracle-evidence settings.

What would settle it

Give human domain experts the same 815 answerable questions and the full HERB retrieval pool, with the same tools or free search, and measure their average score against the paper's 32.96 for the best agent. If experts approach ceiling, the benchmark measures a real gap; if experts also score low, or if inspection shows the synthetic artifacts lack relationships present in actual corporate corpora, the scores would not yet be a reliable statement about enterprise practice.

Watch

Extended reading notes

Core claim

HERB is a heterogeneous enterprise RAG benchmark that requires source-aware, multi-hop reasoning over scattered artifacts. The paper's central claim is that this kind of deep search is meaningfully harder than what current benchmarks measure: even the strongest setup evaluated—a ReAct agent with GPT-4o combining unstructured hybrid retrieval with eight structured lookup tools—reaches an average score of only 32.96, and standard RAG methods range from 10.31 to 20.61. The paper further claims that retrieval is the primary bottleneck, since the same reasoning models perform much better when given complete product-specific context (best long-context average 76.55) or oracle evidence (best average 85.76), though no evaluated model reaches ceiling even in those settings. The release includes both answerable and unanswerable queries so that hallucination and unanswerability detection can be measured alongside accuracy.

Load-bearing premise

HERB's difficulty claims rest on the assumption that LLM-simulated workflows with human-designed distractors faithfully capture real enterprise deep search, so that scores on HERB transfer to practice; the paper itself does not validate against a real enterprise corpus or a human-expert baseline.

Editorial extensions

If this is right

  • None of the standard RAG pipelines—vector, hybrid dense plus BM25, RAPTOR, GraphRAG, HippoRAG-2, or proposition-graph RAG—exceeds an average score of 20.61 on HERB, so current retrieval designs are far from adequate for heterogeneous enterprise deep search.
  • Retrieval, not reasoning, is the limiting factor: Gemini-2.5-Flash scores 76.55 when all product-specific artifacts are given as context but only 41.86 when it must retrieve from the same data, and the gap holds for other strong models.
  • Agentic search helps, but its tool use is shallow: ReAct with GPT-4o beats the best standard baseline by 12.35 points, yet in manual trajectory analysis it often stops at the first retrieved result and sometimes invokes the wrong or unnecessary structured tools.
  • Even with oracle evidence and no retrieval noise, all evaluated LLMs miss points (best average 85.76), meaning faithful reasoning over the correct evidence is itself unsolved.
  • Unanswerable-query detection is unreliable across every system, with best detection at 63.66%; more answer attempts raise accuracy but also raise the hallucination rate on unanswerable questions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: because HERB has no human-expert baseline, its absolute numbers should be treated as a relative difficulty signal; a human ceiling could be well below 100, and the gap between agents and practice may be narrower or wider than it appears.
  • Beyond the paper: the retrieval-bottleneck result suggests that interleaving structured lookups (employee, customer, PR, URL) with re-ranking over heterogeneous sources could yield large gains without a more powerful reasoning model; that is a directly testable design hypothesis on HERB.
  • Beyond the paper: the authors warn against tuning to the specific workflows and distractors, which implies a natural next benchmark should hold out entire workflow families or domains such as finance or healthcare and test whether systems that do well on the released workflows generalize.
  • Beyond the paper: since retrieval recall is implicated, measuring HERB's oracle upper bound per retrieval component—for example, what top-k of each retriever covers of the linked evidence—could turn the qualitative finding into a concrete recall target for future systems.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. HERB is a new benchmark for evaluating deep-search RAG over heterogeneous enterprise data. The authors construct a synthetic enterprise environment with 530 employees, 30 products, and 39,190 artifacts spanning Slack messages, meeting transcripts, documents, GitHub PRs, URLs, and customer profiles. They generate 815 answerable multi-hop queries from manually designed workflows, plus 699 unanswerable queries, and evaluate standard RAG baselines, graph-based retrievers, ReAct agents with structured tools, and long-context LLMs. The best evaluated system, ReAct with GPT-4o, reaches an average score of 32.96; product-specific long-context Gemini-2.5-Flash reaches 76.55; and oracle Gemini-2.5-Flash reaches 85.76. The authors conclude that retrieval is the main bottleneck for current systems.

Significance. If the benchmark is valid, it fills a real gap: existing multi-hop RAG benchmarks often link documents through shallow entity overlap, whereas enterprise deep search requires source-aware reasoning across heterogeneous, noisy artifacts. HERB's construction is detailed, the dataset is released, and several internal controls are well designed: the product-specific long-context versus product-specific RAG comparison isolates retrieval as a variable, the 0-shot baseline scoring 0% on three query types reduces answer-leakage concerns, and the oracle evaluation provides an upper-bound reference. The main weakness is calibration: there is no human expert baseline, no human agreement study for the LLM judge, and no validation against real enterprise corpora. As a result, the absolute scores and the retrieval-bottleneck claim are not yet fully grounded.

major comments (4)
  1. [Section 4.1, Section 5.1] All quantitative results depend on GPT-4o as an evaluator. For content queries, the judge assigns a Likert score from 1 to 100; for the other query types, the judge extracts answers and F1 is computed against ground truth. The paper states in Section 5.1 that the LLM evaluator 'performs consistently with human judgements' but provides no agreement statistic, no sample size, and no description of the human annotation procedure. Without human-judge correlation or inter-annotator agreement, the absolute scores and model comparisons in Tables 2-4 could reflect judge bias, particularly because the judge comes from the same model family (GPT-4o) used to generate the data. Please report agreement on a held-out sample and, where feasible, supplement the LLM judge with human scoring for content queries.
  2. [Section 3.3, Limitations] The headline claim that current systems 'substantially underperform' on HERB is not calibrated against human performance. The Limitations section concedes that the synthetic workflows cover only a small part of how organizations actually work, yet no human expert baseline is provided on HERB queries, and no validation is performed on real enterprise data. The oracle result (Gemini-2.5-Flash at 85.76) shows headroom relative to a perfect system, but not relative to human searchers. A human expert run on a sample of queries, or a separate real-world corpus study, would help determine whether the low absolute scores reflect model deficiency or synthetic-task difficulty. Without such calibration, the external validity of the 32.96 headline and the retrieval-bottleneck conclusion is underdetermined.
  3. [Appendix A.2, Section 4.2] The ReAct agent is defined with a fallback: if an LLM-based quality check deems the response unsatisfactory, the system falls back to the hybrid retriever. The paper does not report the fallback frequency, nor does it ablate the fallback. Because the central improvement claim is the 12.35-point gap between ReAct and the Hybrid baseline, the comparison is not clean unless the fallback is quantified and ReAct-only trajectories are reported separately. Please add an ablation or at least report the fallback rate.
  4. [Section 4.3, Table 3] The retrieval-bottleneck conclusion is drawn from comparing product-specific long-context versus product-specific RAG for Gemini-2.5-Flash (76.55 vs. 41.86) and DeepSeek-R1 (58.66 vs. 34.81). However, GPT-4o shows the opposite pattern, with a higher average in product-specific RAG (43.41) than in long-context (38.84). The text should acknowledge and explain this exception. As written, the claim that 'retrieval limits performance for strong LLMs' appears selective and is not fully supported by the full table.
minor comments (5)
  1. [Abstract, Section 1] The word 'sparsed' should be 'sparse' in the abstract and introduction.
  2. [Table 2] The 0-shot row formats the average and unanswerable scores as '4.5588.70'; inserting a space or separator would improve readability.
  3. [Table 3 caption] The caption contains the typo 'ar eprovided'; it should read 'are provided'.
  4. [Figure 6] The JSON field 'utterranceID' should be spelled 'utteranceID'.
  5. [Appendix A.2] The LLM-based quality check that triggers the hybrid fallback is not specified; please provide the prompting or decision rule used, so readers can assess the fallback's effect.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark is constructed and evaluated empirically, with no derivation chain that reduces to its own inputs.

full rationale

HERB is a benchmark-construction and empirical-evaluation paper, not a mathematical derivation, so the circularity axis applies mainly to the evaluation loop. The ground-truth answers are not simply the generator LLM's outputs: Section 3.3 states that 'to construct an answer, we do not directly use an LLM to generate responses given linked evidence' and instead applies a structured, query-specific inference process over artifacts linked during workflow execution. This breaks the strongest same-family loop. The 0-shot baseline scoring 0% on people-, customer-, and artifact-search queries further indicates that questions are not answerable from parametric memory alone. The content-query scores do rely on a GPT-4o judge with only an asserted, unquantified statement of consistency with human judgments, but structured query types use extraction plus exact-match F1, and no fitted parameter is renamed as a prediction. Self-citations such as CRMArena, UAEval4RAG, SummHay, and PGRAG are used as methodological building blocks or baselines, not as the sole justification for HERB's central claims, and no uniqueness theorem or forbidden-alternative argument rests on them. The Limitations' concession that HERB 'only covers a small part of how organizations actually work' and the absence of a human-expert baseline are external-validity and calibration concerns, not circularity. No step in the paper reduces by construction to its own inputs.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No numbers are fitted to data; the benchmark, not a parameterized model, is the contribution. Configuration choices such as k=20 and attempt budgets are design decisions, not fitted values. No novel physical or conceptual entities are posited; the simulated 530-employee company and its products are data artifacts, not predictions about the world.

assumptions (4)
  • domain assumption LLM-synthesized enterprise data approximates real enterprise data closely enough for benchmark scores to transfer to practice.
    The entire benchmark is generated with LLMs simulating workflows (Section 3.3); the paper provides no comparison to real enterprise corpora, and the Limitations section admits it only covers a small part of how organizations actually work.
  • domain assumption Ground-truth answers, derived by rule-based aggregation over LLM-generated artifacts, are correct.
    Section 3.3 states answers are constructed via structured steps, but the underlying artifacts are synthetic and correctness is not independently human-verified on the released benchmark.
  • domain assumption GPT-4o judgments of content-answer quality are reliable indicators of factual accuracy.
    Section 4.1 uses GPT-4o with Likert ratings; no inter-annotator agreement or human study on the main scores is reported.
  • ad hoc to paper Withholding full workflow specifications prevents users from overfitting to the benchmark.
    Section 3.3 states specifications are available upon request, a policy choice that prevents overfitting but also prevents independent replication of the generation process.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Benchmarking Deep Search over Heterogeneous Enterprise Data." pith.science (2026). https://pith.science/paper/EOIDI3MU

@misc{pith2026250623139,
  author       = {Pith},
  title        = {Pith review of: Benchmarking Deep Search over Heterogeneous Enterprise Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EOIDI3MU}},
  note         = {Machine review of arXiv:2506.23139}
}
read the original abstract

We present a new benchmark for evaluating Deep Search--a realistic and complex form of retrieval-augmented generation (RAG) that requires source-aware, multi-hop reasoning over diverse, sparsed, but related sources. These include documents, meeting transcripts, Slack messages, GitHub, and URLs, which vary in structure and often contain human-to-human interactions. We build it using a synthetic data pipeline that simulates business workflows across product planning, development, and support stages, generating interconnected content with realistic noise and multi-hop questions with guaranteed ground-truth answers. We release our benchmark with both answerable and unanswerable queries, and retrieval pool of 39,190 enterprise artifacts, enabling fine-grained evaluation of long-context LLM and RAG systems. Our experiments reveal that even the best-performing agentic RAG methods achieve an average performance score of 32.96 on our benchmark. With further analysis, we highlight retrieval as the main bottleneck: existing methods struggle to conduct deep searches and retrieve all necessary evidence. Consequently, they often reason over partial context, leading to significant performance degradation.

Figures

Figures reproduced from arXiv: 2506.23139 by the authors.

Figure 1
Figure 1. Overview of the workflow-guided data synthesis framework. Team roles(e.g., Architect) engage through [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Illustration of a product-planning workflow from the HERB dataset. A product manager shares a product [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 4
Figure 4. An example product-related metadata. { "name": "Frank Lewis", "role": "Product Manager", "company": "BlueWave", "products": [ "SupportForce", "VizForce", "WorkflowForce", "SentimentForce", "ForecastForce", "ContentForce" ], "id": "CUST-0001" } [PITH_FULL_IMAGE:figures/full_fig_p016_4.png] view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: An example customer profile. { "Channel": { "name": "planning-SecurityForce-PM", "channelID": "ch-ix-pm-16b727" }, "Message": { "User": { "userId": "eid_c8ebc4b0", "timestamp": "2026-04-28T10:21:00", "text": "Hi team, I've shared the <https://sf-internal.slack.com/ arc…
Figure 6
Figure 6. Figure 6: An example slack message. { "name": "product_dev_WorkFlowGenie_5", "text": "Attendees\nGeorge Garcia, David Garcia, Ian Davis, David Brown, Julia Smith, Alice Taylor, Julia Brown, Charlie Davis, Charlie Martinez, Fiona Martinez, Bob Miller, Julia Garcia, Alice Taylor, …
Figure 7
Figure 7. Figure 7: An example meeting transcript. { "text": "2026-02-11T01:00:00\nHannah Taylor: https://sf-internal. slack.com/archives/docs/final_ectaix_product_vision_document", "id": "ectAIX_planning_2_chat" } [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 9
Figure 9. Figure 9: An example document. { "id": "github_com_LibreOffice_core_pull_1576", "title": "Data Corruption in Calc Reports", "summary": "Generated spreadsheets sometimes contain corrupted summary data, misleading users about financial calculations.", "link": "https://github.com/L…
Figure 10
Figure 10. Figure 10: An example GitHub PR metadata. { "id": "hbr_org_2023_02_the-ethical-implications-of-ai-in-sales", "link": "https://hbr.org/2023/02/the-ethical-implications-of-ai -in-sales", "description": "A Harvard Business Review article discussing the ethical implications of AI in…
Figure 11
Figure 11. Figure 11: An example meeting URL metadata. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. WorkSurface-Bench: Benchmarking Enterprise Agents on Multi-Surface Knowledge Routing

    cs.CL 2026-07 conditional novelty 6.0 of 10

    WorkSurface-Bench measures surface routing separately from answer correctness and finds near-perfect routing still leaves 25–44% answer errors across four LLM backbones.

  2. SFR-DeepResearch: Towards Effective Reinforcement Learning for Autonomously Reasoning Single Agents

    cs.AI 2025-09 conditional novelty 6.0 of 10

    A 20B autonomously reasoning deep-research agent trained with synthetic-data RL reaches 28.7% on Humanity's Last Exam, exceeding several larger and proprietary baselines.

Reference graph

Works this paper leans on

34 extracted references · 8 canonical work pages · cited by 2 Pith papers

  1. [1]

    L \'e o Boisvert, Megh Thakkar, Maxime Gasse, Massimo Caccia, Thibault de Chezelles, Quentin Cappart, Nicolas Chapados, Alexandre Lacoste, and Alexandre Drouin. 2024. Workarena++: Towards compositional planning and reasoning-based common knowledge work tasks. Advances in Neural Information Processing Systems, 37:5996--6051

    work page 2024

  2. [2]

    Prafulla Kumar Choubey, Xin Su, Man Luo, Xiangyu Peng, Caiming Xiong, Tiep Le, Shachar Rosenman, Vasudev Lal, Phil Mui, Ricky Ho, Phillip Howard, and Chien-Sheng Wu. 2024. https://arxiv.org/abs/2410.16597 Distill-synthkg: Distilling knowledge graph synthesis workflow for improved coverage and efficiency . Preprint, arXiv:2410.16597

    arXiv 2024

  3. [3]

    Dvir Cohen, Lin Burg, Sviatoslav Pykhnivskyi, Hagit Gur, Stanislav Kovynov, Olga Atzmon, and Gilad Barkan. 2025. https://arxiv.org/abs/2505.08643 Wixqa: A multi-dataset benchmark for enterprise retrieval-augmented generation . Preprint, arXiv:2505.08643

    arXiv 2025

  4. [4]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 181 others. 2025 a . https://arxiv.org/abs/2501.12948 Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement ...

    arXiv 2025

  5. [5]

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, and 181 others. 2025 b . https://arxiv.org/abs/2412.19437 Deepseek-v3 technical report . Preprint, arXiv:2412.19437

    arXiv 2025

  6. [6]

    Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, Nicolas Chapados, and Alexandre Lacoste

    Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, Nicolas Chapados, and Alexandre Lacoste. 2024. https://arxiv.org/abs/2403.07718 Workarena: How capable are web agents at solving common knowledge work tasks? Preprint, arXiv:2403.07718

    arXiv 2024

  7. [7]

    Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. 2025. https://arxiv.org/abs/2404.16130 From local to global: A graph rag approach to query-focused summarization . Preprint, arXiv:2404.16130

    arXiv 2025

  8. [8]

    ExplodingGradients. 2024. Ragas: Supercharge your llm application evaluations. https://github.com/explodinggradients/ragas

    work page 2024

Show all 34 references
  1. [9]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Art...

    2024 arXiv

  2. [10]

    Bernal Jiménez Gutiérrez, Yiheng Shu, Weijian Qi, Sizhe Zhou, and Yu Su. 2025. https://arxiv.org/abs/2502.14802 From rag to memory: Non-parametric continual learning for large language models . Preprint, arXiv:2502.14802

    2025 arXiv

  3. [11]

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. https://doi.org/10.18653/v1/2020.coling-main.580 Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps . In Proceedings of the 28th International Conference on Computational Li...

    2020 doi

  4. [12]

    Kung-Hsiang Huang, Akshara Prabhakar, Sidharth Dhawan, Yixin Mao, Huan Wang, Silvio Savarese, Caiming Xiong, Philippe Laban, and Chien-Sheng Wu. 2025 a . https://aclanthology.org/2025.naacl-long.194/ CRMA rena: Understanding the capacity of LLM agents to perform professional C...

    2025

  5. [13]

    Kung-Hsiang Huang, Akshara Prabhakar, Onkar Thorat, Divyansh Agarwal, Prafulla Kumar Choubey, Yixin Mao, Silvio Savarese, Caiming Xiong, and Chien-Sheng Wu. 2025 b . Crmarena-pro: Holistic assessment of llm agents across diverse business scenarios and interactions. arXiv prepr...

    2025 arXiv

  6. [14]

    Philippe Laban, Alexander Fabbri, Caiming Xiong, and Chien-Sheng Wu. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.552 Summary of a haystack: A challenge to long-context LLM s and RAG systems . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language ...

    2024 doi

  7. [15]

    Jeongsoo Lee, Daeyong Kwon, Kyohoon Jin, Junnyeong Jeong, Minwoo Sim, and Minwoo Kim. 2025. https://arxiv.org/abs/2504.08756 Mhts: Multi-hop tree structure framework for generating difficulty-controllable qa datasets for rag evaluation . Preprint, arXiv:2504.08756

    2025 arXiv

  8. [16]

    Jerry Liu. 2022. https://doi.org/10.5281/zenodo.1234 LlamaIndex

    2022 doi

  9. [17]

    Meta AI . 2025. The llama 4 herd: The beginning of a new era of natively multimodal ai. https://ai.meta.com/blog/llama-4-multimodal-intelligence/

    2025

  10. [18]

    OpenAI. 2025. Deep research system card. https://cdn.openai.com/deep-research-system-card.pdf

    2025

  11. [19]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

    2024 arXiv

  12. [20]

    Sarah Packowski, Inge Halilovic, Jenifer Schlotfeldt, and Trish Smith. 2024. https://arxiv.org/abs/2410.12812 Optimizing and evaluating enterprise retrieval-augmented generation (rag): A content design perspective . Preprint, arXiv:2410.12812

    2024 arXiv

  13. [22]

    Xiangyu Peng, Prafulla Kumar Choubey, Caiming Xiong, and Chien-Sheng Wu. 2025. https://arxiv.org/abs/2412.12300 Unanswerability evaluation for retrieval augmented generation . Preprint, arXiv:2412.12300

    2025 arXiv

  14. [23]

    Stephen Robertson and Hugo Zaragoza. 2009. https://doi.org/10.1561/1500000019 The probabilistic relevance framework: Bm25 and beyond . Found. Trends Inf. Retr., 3(4):333–389

    2009 doi

  15. [24]

    Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher D. Manning. 2024. https://arxiv.org/abs/2401.18059 Raptor: Recursive abstractive processing for tree-organized retrieval . Preprint, arXiv:2401.18059

    2024 arXiv

  16. [25]

    Yijia Shao, Yucheng Jiang, Theodore A Kanell, Peter Xu, Omar Khattab, and Monica S Lam. 2024. Assisting in writing wikipedia-like articles from scratch with large language models. arXiv preprint arXiv:2402.14207

    2024 arXiv

  17. [26]

    Olly Styles, Sam Miller, Patricio Cerda-Mardini, Tanaya Guha, Victor Sanchez, and Bertie Vidgen. 2024. https://arxiv.org/abs/2405.00823 Workbench: a benchmark dataset for agents in a realistic workplace setting . Preprint, arXiv:2405.00823

    2024 arXiv

  18. [27]

    Yixuan Tang and Yi Yang. 2024. https://arxiv.org/abs/2401.15391 Multihop-rag: Benchmarking retrieval-augmented generation for multi-hop queries . Preprint, arXiv:2401.15391

    2024 arXiv

  19. [28]

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. https://doi.org/10.1162/tacl_a_00475 M u S i Q ue: Multihop questions via single-hop question composition . Transactions of the Association for Computational Linguistics, 10:539--554

    2022 doi

  20. [29]

    Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z

    Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z. Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, Mingyang Yang, Hao Yang Lu, Amaad Martin, Zhe Su, Leander Maben, Raj Mehta, Wayne Chi, Lawrence Jang, Yiqing Xie, and 2 others. 2025. https://arxiv...

    2025 arXiv

  21. [30]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. https://doi.org/10.18653/v1/D18-1259 H otpot QA : A dataset for diverse, explainable multi-hop question answering . In Proceedings of the 2018 Conference...

    2018 doi

  22. [31]

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. 2024. https://arxiv.org/abs/2406.12045 -bench: A benchmark for tool-agent-user interaction in real-world domains . Preprint, arXiv:2406.12045

    2024 arXiv

  23. [32]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. https://arxiv.org/abs/2210.03629 React: Synergizing reasoning and acting in language models . Preprint, arXiv:2210.03629

    2023 arXiv

  24. [33]

    Tan Yu, Wenfei Zhou, Leiyang Leiyang, Aaditya Shukla, Mmadugula Mmadugula, Pritam Gundecha, Nicholas Burnett, Anbang Xu, Viseth Viseth, Tbar Tbar, Rama Akkiraju, and Vivienne Zhang. 2025. https://doi.org/10.18653/v1/2025.knowledgenlp-1.13 EKRAG : Benchmark RAG for enterprise k...

    2025 doi

  25. [34]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  26. [35]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.