Pith. sign in

REVIEW 4 major objections 4 minor 33 references

Production customer-service agents should first diagnose whether the pipeline fails at retrieval, reranking, or final selection, then fix that layer: swapping the LLM backbone changed issue-election by less than 1 percentage point, while a

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 16:15 UTC pith:6YU5TPID

load-bearing objection Solid applied deployment paper with a useful diagnosis-first pattern, but the headline effect sizes are point estimates on tiny holdouts; the paper is honest about this, yet the abstract's takeaway outruns the statistics. the 4 major comments →

arxiv 2607.18039 v1 pith:6YU5TPID submitted 2026-07-20 cs.IR

Evidence-in-the-Loop: Trace-Driven Optimization for Customer-Service LLM Agents

Chunming Wu , Dafei Qiu , Congde Yuan , Charles Quan , Jun Wu , Suipeng Li , Mo Wu , Gavin Xie
show 2 more authors
Hope Chen Max Yao
This is my paper
classification cs.IR
keywords large language modelscustomer serviceretrieval-augmented generationevidence-grounded decision makingrerankingdirect preference optimizationproduction systemstrace-driven optimization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that production customer-service agents should be improved by first diagnosing which layer of the decision pipeline loses the correct answer—retrieval, reranking, or the final LLM's issue/action election—and then applying a targeted update to that layer, rather than by replacing the backbone with a larger or more autonomous model. On a deployed financial customer-service workflow, the authors show that swapping a 27B-class LLM for a frontier model changes final issue-election by less than one percentage point, while a distilled cross-encoder reranker raises held-out top-1 accuracy from 56.76% to 75.68% and sealed 200-session KB-grounded accuracy from 86.5% to 88.5%. Training the final-decision model with preference pairs (DPO) reaches 90.5% on the same holdout, and combining both yields 92.5%. A reader should care because the paper offers a reusable, conservative deployment pattern—evidence-grounded decisions inside a fixed workflow DAG, with trace-driven review feeding reranker hard negatives and decision preference pairs—that keeps LLM autonomy bounded while improving answer quality.

Core claim

The paper's central claim is that in a production customer-service setting, the dominant source of wrong answers is the ordering and election of grounded evidence, not the size of the language model behind the decision. The workflow retrieves FAQ candidates through three parallel channels (BM25, issue-title-vector KNN, issue-description-vector KNN), fuses them with weighted reciprocal rank fusion, rescales them with a cross-encoder reranker, and then lets a constrained LLM choose answer/clarify/handoff from the typed evidence pool. Evidence for the claim: on the 309-query diagnostic subset, hybrid recall reaches 96.76% top-50 coverage while the generic reranker reaches only 56.31% Hit@1, and

What carries the argument

The load-bearing mechanism is the evidence contract: every candidate entering the final decision stage is a typed record carrying a stable FAQ or rule ID, category, provenance, rank signals, and an allowed action surface (Answer, Clarify, Human Handoff). This contract lets a fixed LangGraph DAG fuse general FAQ evidence with scenario-specific rule evidence (from an MCP context layer and a symbolic rule engine) into one comparable pool before the decision LLM elects an issue or action. The second mechanism is the trace-driven improvement loop: replay logs attribute failures to recall, reranking, final election, clarification, or rule evidence, and route them to the right update—KB or query re

Load-bearing premise

The headline 2–4 percentage-point gains over 200 sealed sessions are reported as point estimates with no statistical significance, so if label noise or sampling variance is comparable to the gains, the ordering of configurations is not established—and the decisive reranker split rests on only 37 held-out queries.

What would settle it

Run the same workflow with the same rubric on a new sealed holdout of at least 1,000 sessions, independently blind-labeled by reviewers who did not build the rubric, and require the distilled-reranker (+2pp), DPO (+4pp), and combined (+6pp) gains—as well as the <1pp backbone gap—to persist with confidence intervals excluding zero. A second check: evaluate on original-language (non-machine-translated) user queries in the same domain; if the ordering flips, the paper's diagnosis-first takeaway is not portable.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Diagnosis before upgrade: when a pipeline already has high recall coverage but weak top-1 ranking, reranker and decision adaptation should precede—and can replace—backbone scaling; the less-than-1pp backbone gap is direct evidence.
  • In-domain reranker tuning via teacher-score distillation improves business-domain top-1 while preserving or slightly improving general reranking quality (six-task C-MTEB average from 66.09 to 66.58), and it transfers to session-level gains on a sealed holdout.
  • Decision-stage DPO improves the boundaries between FAQ answers, necessary clarifications, and handoff without changing the evidence path, providing a second independent optimization axis after evidence quality is stable.
  • Combining the distilled reranker with the DPO decision LLM yields the best sealed-session accuracy (92.5%), and a post-launch operational review reports a descriptive 10.5pp gap over a legacy RAG-only workflow under the same session rubric.
  • All session-level gains are measured under a KB-grounded rubric that counts necessary clarification and policy-compliant handoff as correct, not just FAQ ID matching, so the headline numbers reflect production decision quality rather than pure retrieval accuracy.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The evidence contract—typed, provenance-carrying candidates with a bounded action surface—is a candidate design for regulated domains beyond customer service, such as clinical triage or compliance Q&A, where the LLM must be constrained to a small action set; the paper leaves cross-domain transfer untested.
  • A cheap operational heuristic follows from the diagnosis-first method: monitor the gap between recall Hit@50 and reranker Hit@1. If that gap is large, invest in reranker and decision-layer work; if small, invest in retrieval and knowledge-base freshness. This is an extrapolation, not a claim in the paper.
  • Because the headline session numbers are point estimates on 200 sessions with gains of 2–4 percentage points, the ordering of configurations should be re-measured on a larger holdout, ideally with original-language queries (the pipeline machine-translates to English), before other teams copy the recipe.
  • A testable extension is to apply the same trace-driven loop to clarification, rule-evidence, and action metrics—which the paper lists as future work; if those layers dominate failures after launch, the 'reranker-first' conclusion could change by domain.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper reports a deployed customer-service agent workflow in which hybrid retrieval (BM25, issue-title and issue-description vector channels, weighted RRF fusion, cross-encoder reranking) supplies a typed evidence pool to a decision-stage LLM inside a fixed LangGraph DAG, with deterministic guardrails on clarification and handoff. The contribution is a trace-driven improvement loop: failures are attributed to recall, ranking, candidate selection, clarification, or policy, and updates are applied to the corresponding layer. Evaluation is organized as E1 (pipeline diagnosis), E2 (teacher-student reranker fine-tuning), E3 (B-Set session-level validation), E4 (DPO for the final-decision LLM), plus a post-launch operational comparison. The headline claim is that the evidence layer, not backbone scaling, was the main bottleneck, and that reranker adaptation and decision-stage DPO each improve session accuracy.

Significance. If the empirical ordering were robust, the paper would be a useful applied-data-science contribution: it demonstrates a reusable pattern for grounding LLM customer-service decisions, separating evidence construction from decision, and using replay traces to target updates. Strengths include the explicit E1–E4 protocol, the use of a sealed 200-session holdout, the public C-MTEB sanity check to monitor forgetting risk, and the paper's transparency about point estimates and descriptive operational comparison. The main weakness is that the decisive comparisons rest on very small samples and unadjusted point estimates, so the specific ranking of optimization targets is not statistically established. With additional statistical reporting and appropriately hedged claims, the case study remains informative for practitioners.

major comments (4)
  1. [§6.6, Table 11] The central ordering of optimization targets is not statistically established. 86.5% vs 88.5% on N=200 is 173/200 vs 177/200, a change of 4 sessions. Even under the most favorable discordance pattern (4 fail→pass, 0 pass→fail), McNemar's exact test gives two-sided p≈0.125; any reverse discordance increases p. The table caption says 'we do not claim statistical significance', but the abstract and Section 7 use these rows as evidence that reranker adaptation and DPO are the right updates. Please report discordant-pair counts and exact paired confidence intervals (or bootstrap intervals) for all four rows, and soften the headline claims accordingly.
  2. [§6.5, Table 9] Test37 contains 37 queries. Hit@1 improving from 56.76% to 75.68% is 21/37 to 28/37, i.e., 7 additional correct queries; the 95% Wilson confidence intervals overlap. The same issue applies to the supervised row (64.86%). Since the abstract and E2 narrative use this split as the primary evidence for reranker quality, please add an exact paired test or permutation test, or explicitly characterize the result as exploratory. A larger held-out business split would be needed to support the current wording.
  3. [§6.4, Table 8] The backbone-scaling null result is underpowered. Qwen3.5-27B vs GPT-4o gives 81.23% vs 81.88% on N=309 (0.65pp, SE≈2.2pp). This comparison supports only 'no large effect was detected in this diagnostic subset', not the stronger statement in Section 7 that 'backbone scaling alone is not the dominant bottleneck'. Please report a confidence bound on the difference (e.g., a one-sided CI showing the maximum plausible scaling gain) or soften the conclusion to match the evidence.
  4. [§8 and §6.3] The evaluation pipeline machine-translates user utterances to English before retrieval and final decision, so B-Set and T-Set results may not reflect behavior on original-language production traffic. In addition, the KB-grounded review rubric is the authors' own internal gold standard, and no inter-annotator agreement or label-noise analysis is reported. Because the headline gains are 2.0pp (4 sessions on N=200), label noise at a similar scale could change the ordering of configurations. Please report reviewer agreement or a label-noise sensitivity analysis, and explicitly scope the claims to the translated, internally-labeled setting.
minor comments (4)
  1. [Abstract] 'anEvidence-Grounded' should read 'an Evidence-Grounded'.
  2. [§6.3] 'only thelast user turn' is missing a space.
  3. [§6.8] The post-launch operational comparison is properly described as descriptive, but the 10.52pp difference is given without any uncertainty measure; a bootstrap confidence interval would be useful.
  4. [§6.6 and Abstract] The abstract and Section 7 state improvements such as 'raises KB-grounded session accuracy from 86.5% to 88.5%' without the point-estimate caveat that appears in Table 11; the headline text should carry the same hedge as the table caption.

Circularity Check

0 steps flagged

No circularity: predictions are evaluated on held-out Test37/B-Set or external benchmarks; fitted components are not masked as predictions.

full rationale

The paper's derivation chain does not reduce to its inputs. E2 reranker training uses 492 training groups with gold issue IDs and recall-pool hard negatives, selects checkpoints on a 52-query dev split, and reports the headline Hit@1 improvement on a strictly separated 37-query test split (Table 9; Section 6.5). The DPO decision-stage pairs in E4 are explicitly mined from development replay failures and not from the sealed B-Set (Section 6.7), so the 90.5%/92.5% session accuracies are not fit-predictions. RRF weights were chosen by comparing configurations on the 544 non-test T-Set set and are reported as development ablations, not as held-out predictions. B-Set is sealed and reviewed under a fixed rubric, and the same rubric is applied to all compared systems. The only external check is a selected six-task C-MTEB subset, which is a benchmark, not a fitted target. No load-bearing self-citation chain exists: the references to prior work are external tools and benchmarks, not the authors' own results. The paper's own caveats—'Values are point estimates, and we do not claim statistical significance' (Table 11) and the single-domain/translated-query limitations in Section 8—concern statistical power and generalizability, not constructional circularity.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 0 invented entities

The pipeline contributes a deployment structure over well-known RAG components. The RRF formula and retriever/reranker techniques are standard; the paper's load-bearing assumptions are that internal labels and rubrics are reliable, small holdouts are informative, machine translation preserves intent, and separate production entry paths can be compared descriptively. No new physical entities are introduced.

free parameters (5)
  • RRF fusion weights and K = w_BM25=1.05, w_text=0.85, w_desc=1.00, K=40
    Eq. 8; Section 6.4 says this setting reaches the highest T-Set top-50 coverage among tested representative configurations, so the reported Hit@50 on that same non-test set reflects tuned rather than off-the-shelf choices.
  • Retrieval fetch sizes and pool/window sizes = BM25 fetch 150, vector fetch 10/10, N_RRF=50, decision window 20 then 10
    Sections 4.6 and 6.5: values chosen by ablation on T-Set; the top-10 serving window is justified after adaptation using Test37 and B-Set.
  • Reranker training data cleaning threshold = 14 of 506 candidate groups removed
    Section 6.5 removes groups with low teacher margin, mismatch, or weak labels; this post-hoc selection can inflate E2 gains.
  • DPO preference-pair selection = 117 pairs (53 wrong-issue-to-clarify, 38 wrong-issue-to-issue, 26 wrong-clarify-to-issue)
    Section 6.7: pairs are mined from development replay failures; the confidence threshold/count is a human-chosen parameter.
  • Teacher-score distillation weight
    E2 uses a teacher-score distillation term but the loss weight is not reported; replication would require choosing this hyperparameter.
axioms (6)
  • standard math Reciprocal rank fusion (Eq. 8) with fixed K=40 and channel weights yields a valid ranking for downstream reranking and selection.
    Used as a background retrieval technique from refs [3,4,6]; not derived in this paper.
  • domain assumption The T-Set gold FAQ issue IDs, human+AI review outputs, and B-Set KB-grounded review rubric measure the 'correctness' the paper claims.
    Sections 5, 6.3, and 6.6: all headline claims rest on these internal labels; no inter-annotator agreement or external validation is reported.
  • domain assumption Machine-translated English queries preserve user intent for retrieval and final decision.
    Sections 6.3 and 8: user utterances are machine-translated to English; translation errors may affect recall and decision quality.
  • domain assumption The passage 'values are point estimates, and we do not claim statistical significance' is the appropriate error model for B-Set comparisons.
    Table 11 and Figure 7 rely on 200 sessions without confidence intervals; inferences about 2pp gains assume differences exceed evaluation noise.
  • domain assumption The selected six-task C-MTEB proxy captures general reranking forgetting risk.
    Section 6.5 and Table 10: a subset is chosen to monitor forgetting; no full C-MTEB or BEIR sweep is reported.
  • domain assumption Legacy and Agent production samples are comparable except for workflow; the post-launch 10.52pp difference is descriptive, not causal.
    Section 6.8 acknowledges samples come from separate entry paths without randomized assignment or matching.

pith-pipeline@v1.3.0-alltime-deepseek · 18713 in / 12998 out tokens · 128933 ms · 2026-08-01T16:15:18.585448+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Evidence-in-the-Loop: Trace-Driven Optimization for Customer-Service LLM Agents." pith.science (2026). https://pith.science/paper/6YU5TPID

@misc{pith2026260718039,
  author       = {Pith},
  title        = {Pith review of: Evidence-in-the-Loop: Trace-Driven Optimization for Customer-Service LLM Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6YU5TPID}},
  note         = {Machine review of arXiv:2607.18039}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Production customer-service bots must improve answer quality across iterative releases, yet large language models must not bypass evidence boundaries, policy rules, or human-handoff safeguards. We present an \textbf{Evidence-Grounded Customer-Service Agent Workflow} deployed in a real-world customer-service setting. BM25 recall, issue-title-vector recall, issue-description-vector recall, weighted RRF fusion, and cross-encoder reranking construct grounded FAQ evidence for controlled LLM decisions. Policy-guided orchestration then combines this RAG evidence with scenario-specific rule evidence, conversation memory, and clarification state inside a fixed LangGraph DAG~\cite{langgraph2024}. The paper contributes three reusable deployment patterns: \textbf{hybrid RAG evidence construction}, where multi-channel retrieval and reranking produce auditable FAQ candidates; \textbf{evidence-grounded issue/action decision}, where an Evidence-Grounded Decision Module selects an issue/action from typed FAQ evidence and scenario-specific rule evidence; and \textbf{trace-driven RAG and reranker improvement}, where traces diagnose whether failures come from recall, ranking, final candidate selection, clarification, rule-derived evidence, or action policy, and where reranker fine-tuning is evaluated not only for in-domain gain but also for forgetting risk.

Figures

Figures reproduced from arXiv: 2607.18039 by Charles Quan, Chunming Wu, Congde Yuan, Dafei Qiu, Gavin Xie, Hope Chen, Jun Wu, Max Yao, Mo Wu, Suipeng Li.

Figure 1
Figure 1. Figure 1: Trace-driven review and improvement loop. Batch replay surfaces failures with retrieval evidence; Human+AI review [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Three-layer logical architecture: client channel, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 5
Figure 5. Figure 5: Three-channel hybrid FAQ recall with weighted [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Anonymized support-agent workspace with evidence annotation. Support agents review the masked user query, [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: B-Set session accuracy under a two-axis optimiza [PITH_FULL_IMAGE:figures/full_fig_p011_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

33 extracted references · 2 canonical work pages

  1. [1]

    Anthropic PBC. 2024. Model Context Protocol. https://modelcontextprotocol.io/ docs/getting-started/intro. Accessed June 2026. 12 Evidence-in-the-Loop: Trace-Driven Optimization for Customer-Service LLM Agents

    2024

  2. [2]

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. InInternational Conference on Learning Representations. https://openreview.net/ forum?id=hSyW5go0v8

    2024

  3. [3]

    Sebastian Bruch, Siyu Gai, and Amir Ingber. 2024. An Analysis of Fusion Func- tions for Hybrid Retrieval.ACM Transactions on Information Systems42, 1, Article 20 (2024), 20:1–20:35 pages. doi:10.1145/3596512

    doi:10.1145/3596512 2024

  4. [4]

    Tao Chen, Mingyang Zhang, Jing Lu, Michael Bendersky, and Marc Najork. 2022. Out-of-Domain Semantics to the Rescue! Zero-Shot Hybrid Retrieval Models. In Advances in Information Retrieval: 44th European Conference on IR Research, ECIR 2022 (Lecture Notes in Computer Science, Vol. 13185). 95–110. doi:10.1007/978-3- 030-99736-6_7

    doi:10.1007/978-3- 2022

  5. [5]

    Willy Chung, Samuel Cahyawijaya, Bryan Wilie, Holy Lovenia, and Pascale Fung

  6. [6]

    Cormack, Charles L

    Gordon V. Cormack, Charles L. A. Clarke, and Stefan Buettcher. 2009. Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods. InProceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval. 758–759. doi:10.1145/1571941.1572114

    arXiv 2009

  7. [7]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2024. Retrieval-Augmented Gen- eration for Large Language Models: A Survey.arXiv preprint arXiv:2312.10997 (2024). doi:10.48550/arXiv.2312.10997

    Pith/arXiv arXiv doi:10.48550/arxiv.2312.10997 2024

  8. [8]

    Artur d’Avila Garcez and Luis C. Lamb. 2023. Neurosymbolic AI: The 3rd Wave. Artificial Intelligence Review56, 11 (2023), 12387–12406. doi:10.1007/s10462-023- 10448-w

    doi:10.1007/s10462-023- 2023

  9. [9]

    Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong Park

  10. [10]

    LangChain AI. 2024. LangGraph: Build Stateful, Multi-Actor Applications with LLMs. https://langchain-ai.github.io/langgraph/. Accessed June 2026

    2024

  11. [11]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. InAdvances in Neural Infor- mation Processing Systems, Vol. 33. 9459–9474. https://proceedings....

    2020

  12. [12]

    Microsoft. 2026. Azure AI Search: Hybrid Search. https://learn.microsoft.com/en- us/azure/search/hybrid-search-overview. Accessed 2026-06-16

    2026

  13. [13]

    Microsoft. 2026. Azure AI Search: Reciprocal Rank Fusion. https://learn.microsoft. com/en-us/azure/search/hybrid-search-ranking. Accessed 2026-06-16

    2026

  14. [14]

    Microsoft. 2026. Azure AI Search: Semantic Ranking. https://learn.microsoft. com/en-us/azure/search/semantic-search-overview. Accessed 2026-06-16

    2026

  15. [15]

    Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage Re-ranking with BERT. arXiv preprint arXiv:1901.04085(2019). doi:10.48550/arXiv.1901.04085

    Pith/arXiv arXiv doi:10.48550/arxiv.1901.04085 2019

  16. [16]

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2024. ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs. InInternational Conference on Learning Repre...

    2024

  17. [17]

    Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 3982–3992. doi:10.18653/v1/ D19-1410

    doi:10.18653/v1/ 2019

  18. [18]

    Stephen Robertson and Hugo Zaragoza. 2009. The Probabilistic Relevance Frame- work: BM25 and Beyond.Foundations and Trends in Information Retrieval3, 4 (2009), 333–389. doi:10.1561/1500000019

    doi:10.1561/1500000019 2009

  19. [19]

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessí, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicolas Cancedda, and Thomas Scialom. 2023. Toolformer: Language Models Can Teach Themselves to Use Tools. InAdvances in Neural Information Processing Systems, Vol. 36. https://arxiv.org/abs/2302.04761

    Pith/arXiv arXiv 2023

  20. [20]

    Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language Agents with Verbal Reinforcement Learning.arXiv preprint arXiv:2303.11366(2023). doi:10.48550/ arXiv.2303.11366

    Pith/arXiv arXiv doi:10.48550/arxiv.2303.11366 2023

  21. [21]

    Vasilakos

    Aditi Singh, Abul Ehtesham, Saket Kumar, Tala Talaei Khoei, and Athanasios V. Vasilakos. 2025. Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG.arXiv preprint arXiv:2501.09136(2025). doi:10.48550/arXiv.2501.09136

    Pith/arXiv arXiv doi:10.48550/arxiv.2501.09136 2025

  22. [22]

    Yiyou Sun, Junjie Hu, Wei Cheng, and Haifeng Chen. 2024. DFA-RAG: Con- versational Semantic Router for Large Language Model with Definite Finite Automaton. InProceedings of the 41st International Conference on Machine Learn- ing (Proceedings of Machine Learning Research, Vol. 235). PMLR, 47033–47055. https://proceedings.mlr.press/v235/sun24e.html

    2024

  23. [23]

    Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. InProceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, Vol. 1. https://openreview.net/forum? id=wCu6T5xFjeJ

    2021

  24. [24]

    Rojas- Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young

    Tsung-Hsien Wen, David Vandyke, Nikola Mrkšić, Milica Gašić, Lina M. Rojas- Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017. A Network-based End-to-End Trainable Task-Oriented Dialogue System. InProceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers. 438–449. https://acl...

    2017

  25. [25]

    Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. 2024. C-Pack: Packed Resources For General Chinese Embeddings. InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 641–649. doi:10.1145/3626772.3657878 Accessed June 2026

    arXiv 2024

  26. [26]

    Zhentao Xu, Mark Jerome Cruz, Matthew Guevara, Tie Wang, Manasi Deshpande, Xiaofeng Wang, and Zheng Li. 2024. Retrieval-Augmented Generation with Knowledge Graphs for Customer Service Question Answering. InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2905–2909. doi:10.1145/3626772.3661370

    arXiv 2024

  27. [27]

    Shi-Qi Yan, Jia-Chen Gu, Yun Zhu, and Zhen-Hua Ling. 2024. Corrective Retrieval Augmented Generation.arXiv preprint arXiv:2401.15884(2024). doi:10.48550/ arXiv.2401.15884

    Pith/arXiv arXiv doi:10.48550/arxiv.2401.15884 2024

  28. [28]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Mod- els. InInternational Conference on Learning Representations. https://openreview. net/forum?id=WE_vluYUL-X

    2023

  29. [29]

    Williams

    Steve Young, Milica Gašić, Blaise Thomson, and Jason D. Williams. 2013. POMDP- Based Statistical Spoken Dialog Systems: A Review.Proc. IEEE101, 5 (2013), 1160–1179. doi:10.1109/JPROC.2012.2225812

    arXiv 2013

  30. [30]

    Cen Zhao, Tiantian Zhang, Hanchen Su, Yufeng Zhang, Shaowei Su, Mingzhi Xu, Yu Liu, Wei Han, Jeremy Werner, Claire Na Cheng, and Yashar Mehdad. 2025. Agent-in-the-Loop: A Data Flywheel for Continuous Improvement in LLM-based Customer Support. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track. Associat...

    work page doi:10.18653/v1/2025.emnlp-industry.135 2025

  31. [31]

    Chenyu Zhou, Huacan Chai, Wenteng Chen, Zihan Guo, Rong Shan, Yuanyi Song, Tianyi Xu, Yingxuan Yang, Aofan Yu, Weiming Zhang, Congming Zheng, Jiachen Zhu, Zeyu Zheng, Zhuosheng Zhang, Xingyu Lou, Changwang Zhang, Zhihui Fu, Jun Wang, Weiwen Liu, Jianghao Lin, and Weinan Zhang. 2026. Externalization in LLM Agents: A Unified Review of Memory, Skills, Protoc...

    Pith/arXiv arXiv doi:10.48550/arxiv.2604 2026

  32. [2023]

    InProceedings of the Second Workshop on Natural Language Interfaces

    InstructTODS: Large Language Models for End-to-End Task-Oriented Dialogue Systems. InProceedings of the Second Workshop on Natural Language Interfaces. 1–21. doi:10.18653/v1/2023.nlint-1.1

    work page doi:10.18653/v1/2023.nlint-1.1 2023

  33. [2024]

    InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)

    Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 7036–7050. doi:10.18653/v1/2024. naacl-long.389

    doi:10.18653/v1/2024 2024