Pith. sign in

REVIEW 5 major objections 6 minor 92 references

From Unstructured Communication to Intelligent RAG: Multi-Agent Automation for Supply Chain Knowledge Bases

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A three-agent pipeline that converts raw support tickets into a category-structured knowledge base before retrieval raises RAG helpful answers from 38.60% to 48.74% and cuts unhelpful answers by 77.4%.

desk verdict A practical offline-first RAG pipeline with a genuinely useful engineering story, but the headline 48.74% vs 38.60% gap is judged by the same model family that wrote the knowledge base and the test queries, so the numbers should be treated as provisional until a human or cross-family check. read the letter →

arxiv 2506.17484 v1 pith:Y7SEKL5I submitted 2025-06-20 cs.AI

Yao Zhang , Zaixi Shang , Silpan Patel , Mikel Zuniga This is my paper
classification cs.AI
keywords retrieval-augmentedgenerationmulti-agentsystemsknowledgebaseconstructionsupplychainoperationssupportticketanalysiscategorydiscoverysynthesisoffline-firstmethodology
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the raw tickets themselves, not the retriever or generator, are the main bottleneck in retrieval-augmented question answering over support communications. It proposes an offline-first pipeline in which three specialized LLM agents discover a category taxonomy, assign tickets to categories, and synthesize grouped tickets into concise knowledge articles. On a real-world supply chain ticket corpus, this knowledge base contains only 3.4% of the original ticket volume, yet raises the share of answers judged helpful from 38.60% to 48.74% and reduces completely unhelpful answers by 77.4%. If the result holds, institutional knowledge captured in tickets can be made reusable automatically, without manual documentation and without adding query-time latency or cost.

What carries the argument

The central mechanism is multi-agent category-driven synthesis, an offline pipeline that turns noisy tickets into structured articles before any query arrives. Category Discovery builds a taxonomy from sampled tickets; Ticket Categorization assigns each ticket to up to two categories; Knowledge Synthesis converts grouped tickets into 400-to-500-word articles covering common issues, proven resolutions, and resources. Large categories are split into subcategories and synthesized hierarchically, and batch-produced articles are merged with a dedicated merge prompt. The same synthesis prompt is used throughout so the knowledge base stays coherent while its volume drops to 3.4% of the original ticket data.

What would settle it

Have human support resolvers score the same answers for whether they would let a ticket be closed and compare their rankings to the LLM judge's scores; if human resolution rates do not track the judge, or the 48.74% versus 38.60% gap narrows under human scoring, the central claim would have to be revised.

Watch

Extended reading notes

Core claim

The central discovery is that cleaning and restructuring the knowledge base before retrieval can improve end-to-end RAG answer quality more than the raw corpus can. The authors report that their multi-agent category-driven synthesis pipeline, which processes tickets in the order category discovery, ticket categorization, and knowledge synthesis, produces a compact knowledge base that outperforms raw ticket indexing, single-ticket synthesis, and embedding-based clustering aggregation on helpfulness. The measured gain is 3.43 versus 3.08 mean helpfulness and 48.74% versus 38.60% helpful answers, with a 77.4% reduction in score-1 responses and statistical significance across three runs. The authors further estimate that this level of answer quality could help resolve roughly half of future supply chain tickets.

Load-bearing premise

The result stands on the assumption that an LLM judge rating answers against the original ticket measures real-world usefulness, and that queries generated from historical ticket titles and descriptions represent the questions future users will ask.

Editorial extensions

If this is right

  • Knowledge-base construction can be moved offline, so the expensive synthesis work does not add latency or cost to query processing.
  • A knowledge base at 3.4% of the original volume can outperform the full raw corpus, implying large savings in storage, retrieval footprint, and maintenance.
  • The offline approach complements runtime RAG optimizations, so organizations can combine preprocessing synthesis with online retrieval improvements.
  • With nearly half of test queries receiving helpful answers and the worst-answer rate falling sharply, a substantial share of recurring tickets could be handled without a human resolver.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because test queries are rebuilt from historical ticket titles and descriptions, the 48.74% helpful-answer rate may not extend to genuinely new problem types that have no analogue in the training period.
  • Editorial inference: the same discover-categorize-synthesize pipeline should transfer to other unstructured operational corpora, such as incident reports or engineering logs, wherever a data-driven taxonomy can be induced.
  • Editorial inference: an ablation that varies the number of retrieved articles per method could separate the benefit of synthesis from the benefit of a smaller, denser retrieval window, since the competing methods are compared with different corpus sizes and top-k settings.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes an offline-first multi-agent framework that converts unstructured supply-chain support tickets into a structured knowledge base. Three agents—Category Discovery, Categorization, and Knowledge Synthesis—build a taxonomy, assign tickets to categories, and generate short knowledge articles, which are then used as the retrieval corpus for a RAG system. On an internal Amazon ticket dataset, the authors report that their knowledge base (3.4% of original volume) yields 48.74% 'helpful' answers versus 38.60% for raw-ticket indexing, a 77.4% reduction in unhelpful responses, and they infer that roughly 50% of future tickets could be resolved automatically. The paper also compares against single-agent synthesis, embedding-based clustering, and a multi-level integration approach.

Significance. The contribution is practically relevant: an offline knowledge-construction step is a sensible complement to runtime RAG optimization, and the multi-agent taxonomy-driven synthesis is clearly described with reproducible prompts. If the measured improvement is real, the method would be valuable for enterprise support knowledge management. However, the central quantitative claim depends on a single LLM judge from the same model family that generated both the knowledge articles and the test queries, with no human evaluation; this makes the headline gain uncertain. The paper's strengths are the detailed methodology and evaluation transparency; its main weakness is the validity of the primary metric.

major comments (5)
  1. [Section 4.3, Appendix C] The primary comparison is evaluated exclusively by Claude Sonnet 3.7 helpfulness ratings, while the knowledge articles are synthesized by Claude Sonnet 3.7 (Appendix B.6) and test queries are generated by Claude Sonnet 3.7 (Appendix D). This means the judge, generator, and query generator are from the same model family, so the 10.14-percentage-point gap could partly reflect stylistic self-preference rather than resolution quality. Because no human evaluation or cross-family judge is reported, the manuscript's central claim that the prebuilt knowledge base 'significantly outperforms' raw-ticket RAG is not yet established. I recommend adding a human expert evaluation on a sampled set of answers, or at least a second judge from a different model family with agreement statistics.
  2. [Section 5.3] The claim of 'p < 0.001' is not adequately supported. The text says the analysis used 'the results across the three evaluation runs', i.e., three data points per condition. With n=3 per group, a Welch's t-test has only a few degrees of freedom, and the reported p-value requires exact test statistics and degrees of freedom. If instead the test was performed over individual answers, the independence assumption is violated because answers within a run share retrieval and generation context. Please report the unit of analysis, the test statistic, degrees of freedom, and a confidence interval for the difference.
  3. [Abstract and Section 5.3] The abstract states a '77.4% reduction in unhelpful responses', but Section 5.3 reports a 43.2% reduction for scores 1-2 and a 77.4% reduction for score=1 only. These are different claims; the headline number should use the same definition as the main result. If the primary claim is about all unhelpful responses (scores 1-2), the 77.4% figure should be replaced by 43.2%, or explicitly labeled as score=1 responses.
  4. [Section 4.2, method (5)] Multi-Level Knowledge Integration retrieves top 5 articles from each of the two knowledge bases, whereas all other methods retrieve top 10. This asymmetry makes it difficult to interpret why multi-level integration did not outperform the proposed approach; the comparison would be cleaner with the same total retrieval budget (e.g., top 5+5 vs top 10 of a combined index) or with retrieval ablations.
  5. [Section 5.4 and Conclusion] The statement that the system can 'automatically resolve approximately 50% of future tickets' extrapolates from the percentage of helpful answers to LLM-generated queries on held-out tickets. Helpfulness rated by an LLM is not the same as actual ticket resolution; without a deployment study or resolver-confirmed resolution outcome, this operational claim is not supported.
minor comments (6)
  1. [Tables 1 and 3] The column 'Knowledge Articles (%)' is not defined; clarify whether it is the share of article count or token/character volume relative to raw tickets.
  2. [Section 4.1] The sentence 'we do not report absolute metrics on internal data and instead report all results relative to a baseline' conflicts with Table 1 reporting absolute average helpfulness scores (3.08, 3.43); clarify what 'relative' means here.
  3. [Appendix D] The query generation prompt is described, but no examples of generated queries are shown; a few examples would help assess whether the queries resemble real user requests.
  4. [Section 5.2] The relative percentage shifts (e.g., +66.1% for score=4) are hard to assess without baseline percentages; include the underlying score distribution or absolute percentages.
  5. [Section 2.1] The sentence 'A summary of related work is in the Appendix. Table 2.' is a formatting artifact; fix the punctuation and integration of the table reference.
  6. [References] Some references (e.g., [1], [12], [17]) are non-archival blog posts; replacing them with peer-reviewed or archival sources would strengthen the related-work discussion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the multi-agent KB is built from the training split and judged on held-out tickets, with no fitted parameter; the same-model LLM evaluation is a measurement-validity threat, not a circular derivation.

full rationale

The derivation chain is not circular. The knowledge base is constructed exclusively from the chronological training split (Section 4.1) and evaluated on held-out validation/test tickets; no parameter is fitted to the helpfulness scores. The three-agent pipeline (category discovery, categorization, synthesis) is a fixed LLM-prompt procedure with no feedback from the evaluation metric, so the reported 48.74% versus 38.60% gap is an empirical outcome rather than a construction identity. The abstract's 77.4% unhelpful-response reduction is an internal inconsistency with Section 5.3's 43.2% figure for scores 1-2, and the fact that Claude Sonnet 3.7 generates the knowledge-base articles, the test queries, and the helpfulness ratings is a legitimate measurement-validity threat (possible self-preference), but neither reduces the central comparison to its inputs by definition. The compactness (3.4% of ticket volume) is partly enforced by the synthesis prompt's explicit 400-500-word limit, but the paper does not present that ratio as a fitted prediction; it is a designed property of the method. No load-bearing self-citations or imported uniqueness theorems appear. Therefore, no enumerated circularity step is present.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central result is an engineering evaluation, not a derivation. Its validity rests on assumptions about ticket data sufficiency, temporal stability, query simulation, and the reliability of an LLM judge. These are not derived or externally validated in the paper, and the proprietary data prevents direct inspection of how well the knowledge articles preserve the source content.

free parameters (4)
  • Category size threshold for batch synthesis = 10 (default)
    Below this threshold, all tickets in a category are synthesized in a single pass; the value is configurable and affects article granularity and retrieval quality.
  • Category size threshold for hierarchical subcategory synthesis = 50 (default)
    Above this threshold, categories are split into subcategories before synthesis, changing the structure and article count of the knowledge base.
  • Maximum categories assigned per ticket = 2
    Each ticket can be assigned to up to two categories, which determines grouping and downstream synthesis.
  • Retrieval top-k = 10 (5+5 for the multi-level method)
    The number of knowledge articles retrieved per query directly affects RAG answer quality and is not held constant across all compared methods: the multi-level approach retrieves 5 from each of two knowledge bases.
assumptions (4)
  • domain assumption Support tickets with resolution notes and comments contain sufficient explicit information to resolve a large fraction of future tickets.
    The entire approach is premised on tickets being a reliable source of generalizable operational knowledge; the authors state this in Section 3.1 without empirical support beyond the reported end-to-end results.
  • domain assumption A chronological train/validation/test split prevents leakage and reflects real-world knowledge base creation.
    Stated in Section 4.1; assumes the distribution of ticket types is stable enough that a knowledge base built on past tickets will help on future tickets.
  • ad hoc to paper LLM-generated queries from ticket titles and descriptions, excluding comments, are representative of real user support queries.
    Described in Section 4.2 and Appendix D; the query generation is a simulation and has no validation against actual user queries.
  • ad hoc to paper Claude Sonnet 3.7's helpfulness ratings against the source ticket are a valid measure of real-world answer helpfulness.
    Section 4.3 and Appendix C; no human evaluation or calibration is provided, and the judge is from the same model family as the system being evaluated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Unstructured Communication to Intelligent RAG: Multi-Agent Automation for Supply Chain Knowledge Bases." pith.science (2026). https://pith.science/paper/Y7SEKL5I

@misc{pith2026250617484,
  author       = {Pith},
  title        = {Pith review of: From Unstructured Communication to Intelligent RAG: Multi-Agent Automation for Supply Chain Knowledge Bases},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y7SEKL5I}},
  note         = {Machine review of arXiv:2506.17484}
}
read the original abstract

Supply chain operations generate vast amounts of operational data; however, critical knowledge such as system usage practices, troubleshooting workflows, and resolution techniques often remains buried within unstructured communications like support tickets, emails, and chat logs. While RAG systems aim to leverage such communications as a knowledge base, their effectiveness is limited by raw data challenges: support tickets are typically noisy, inconsistent, and incomplete, making direct retrieval suboptimal. Unlike existing RAG approaches that focus on runtime optimization, we introduce a novel offline-first methodology that transforms these communications into a structured knowledge base. Our key innovation is a LLMs-based multi-agent system orchestrating three specialized agents: Category Discovery for taxonomy creation, Categorization for ticket grouping, and Knowledge Synthesis for article generation. Applying our methodology to real-world support tickets with resolution notes and comments, our system creates a compact knowledge base - reducing total volume to just 3.4% of original ticket data while improving quality. Experiments demonstrate that our prebuilt knowledge base in RAG systems significantly outperforms traditional RAG implementations (48.74% vs. 38.60% helpful answers) and achieves a 77.4% reduction in unhelpful responses. By automating institutional knowledge capture that typically remains siloed in experts' heads, our solution translates to substantial operational efficiency: reducing support workload, accelerating resolution times, and creating self-improving systems that automatically resolve approximately 50% of future supply chain tickets. Our approach addresses a key gap in knowledge management by transforming transient communications into structured, reusable knowledge through intelligent offline processing rather than latency-inducing runtime architectures.

Figures

Figures reproduced from arXiv: 2506.17484 by the authors.

Figure 1
Figure 1. Multi-Agent Category-Driven Synthesis Architec [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Score Distribution Comparison Across Knowledge [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

92 extracted references · 70 canonical work pages

  1. [1]

    Anthony Alcaraz. 2024. Enterprise RAG at Scale: How to unlock Value with Optimized Architectures and Responsible AI? Medium blog post. https://medium.com/codex/enterprise-rag-at-scale-how-to-unlock-value- with-optimized-architectures-and-responsible-ai-9e848597c0fb

    2024

  2. [2]

    Sara AlMahri, Liming Xu, and Alexandra Brintrup. 2024. Enhancing Supply Chain Visibility with Knowledge Graphs and Large Language Models. doi:10. 48550/arXiv.2408.07705 arXiv:2408.07705 [cs] version: 1

    arXiv doi:10.48550/arxiv.2408.07705 2024

  3. [3]

    Amazon Web Services. 2024. Amazon Titan Text Embeddings V2. https://docs.aws. amazon.com/bedrock/latest/userguide/titan-embedding-models.html. Accessed: April 27, 2025

    2024

  4. [4]

    Eric Anderson, Jonathan Fritz, Austin Lee, Bohou Li, Mark Lindblad, Henry Lin- deman, Alex Meyer, Parthkumar Parmar, Tanvi Ranade, Mehul A Shah, Benjamin Sowell, Dan Tecuci, Vinayak Thapliyal, and Matt Welsh. 2025. The Design of an LLM-powered Unstructured Analytics System. (2025)

    2025

  5. [5]

    Ricardo JGB Campello, Davoud Moulavi, and Jörg Sander. 2013. Density-based clustering based on hierarchical density estimates.Advances in Knowledge Dis- covery and Data Mining(2013), 160–172

    2013

  6. [6]

    Shuaihang Chen, Yuanxing Liu, Wei Han, Weinan Zhang, and Ting Liu. 2025. A Survey on LLM-based Multi-Agent System: Recent Advances and New Frontiers in Application. doi:10.48550/arXiv.2412.17481 arXiv:2412.17481 [cs]

    arXiv doi:10.48550/arxiv.2412.17481 2025

  7. [7]

    Pathway Community. 2025. Multi Agent RAG with Interleaved Retrieval and Rea- soning for Long Docs. https://pathway.com/blog/multi-agent-rag-interleaved- retrieval-reasoning

    2025

  8. [8]

    Duffy, Iman Evazzade, Torsten Foehr, Gregory Robbins, Debbrata Kumar Saha, Jiya Varma, and Marcin Ziolkowski

    Noel Crawford, Edward B. Duffy, Iman Evazzade, Torsten Foehr, Gregory Robbins, Debbrata Kumar Saha, Jiya Varma, and Marcin Ziolkowski. 2024. BMW Agents – A Framework For Task Automation Through Multi-Agent Collaboration. doi:10. 48550/arXiv.2406.20041 arXiv:2406.20041 [cs]

    arXiv doi:10.48550/arxiv.2406.20041 2024

Show all 92 references
  1. [9]

    Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. 2025. From Local to Global: A Graph RAG Approach to Query-Focused Summarization. doi:10.48550/arXiv.2404.16130 arXiv:240...

    2025 doi arXiv

  2. [10]

    Chawla, Olaf Wiest, and Xiangliang Zhang

    Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V. Chawla, Olaf Wiest, and Xiangliang Zhang. 2024. Large Language Model based Multi-Agents: A Survey of Progress and Challenges. doi:10.48550/arXiv.2402. 01680 arXiv:2402.01680 [cs]

    2024 doi arXiv

  3. [11]

    Lionel Richy Panlap Houamegni and Fatih Gedikli. 2025. Evaluating the Effec- tiveness of Large Language Models in Automated News Article Summarization. doi:10.48550/arXiv.2502.17136 arXiv:2502.17136 [cs]

    2025 doi arXiv

  4. [12]

    Abhinav Kimothi. 2024. RAGOps Guide: Building and Scaling Retrieval Augmented Generation Systems. Medium (Towards Data Science) blog post. https://medium.com/data-science/ragops-guide-building-and-scaling- retrieval-augmented-generation-systems-3d26b3ebd627

    2024

  5. [13]

    Rajeev Kumar, Kumar Ishan, Harishankar Kumar, and Abhinandan Singla. 2025. LLM-Powered Knowledge Graphs for Enterprise Intelligence and Analytics. doi:10.48550/arXiv.2503.07993 arXiv:2503.07993 [cs]

    2025 doi arXiv

  6. [14]

    Weitao Li, Kaiming Liu, Xiangyu Zhang, Xuanyu Lei, Weizhi Ma, and Yang Liu

  7. [15]

    Jerry Liu. 2024. Beyond chatbots: adopting Agentic Document Workflows for enterprises — LlamaIndex - Build Knowledge Assistants over your Enter- prise Data. https://www.llamaindex.ai/blog/beyond-chatbots-adopting-agentic- document-workflows-for-enterprises

    2024

  8. [16]

    1995.The Knowledge-Creating Company: How Japanese Companies Create the Dynamics of Innovation

    Ikujiro Nonaka and Hirotaka Takeuchi. 1995.The Knowledge-Creating Company: How Japanese Companies Create the Dynamics of Innovation. Oxford University Press, New York, NY. Introduces the SECI model (Socialization, Externalization, Combination, Internalization) of knowledge cre...

    1995

  9. [17]

    Polymer Solutions, Inc. 2025. Introducing Polymer’s SecureRAG: AI-powered data security for rapid & secure deployment of LLM agents in enterprises. Polymer blog post. https://www.polymerhq.io/blog/introducing-polymers- securerag-ai-powered-data-security-for-rapid-secure-deploy...

    2025

  10. [18]

    Yongqian Sun, Bowen Hao, Xiaotian Wang, Chenyu Zhao, Yongxin Zhao, Binpeng Shi, Shenglin Zhang, Qiao Ge, Wenhu Li, Hua Wei, and Dan Pei. 2025. LLM- Augmented Ticket Aggregation for Low-cost Mobile OS Defect Resolution. In Companion Proceedings of the 33rd ACM Symposium on the ...

    2025

  11. [19]

    Paulina Toro Isaza, Michael Nidd, Noah Zheutlin, Jae-wook Ahn, Chi- dansh Amitkumar Bhatt, Yu Deng, Ruchi Mahindru, Martin Franz, Hans Florian, and Salim Roukos. 2024. Retrieval Augmented Generation-Based Incident Reso- lution Recommendation System for IT Support.arXiv preprin...

    2024 arXiv

  12. [20]

    Suyuan Wang, Xueqian Yin, Menghao Wang, Ruofeng Guo, and Kai Nan. 2024. EvoPat: A Multi-LLM-based Patents Summarization and Analysis Agent. doi:10. 48550/arXiv.2412.18100 arXiv:2412.18100 [cs] version: 1

    2024 doi arXiv

  13. [21]

    Jochen Wulf and Jürg Meierhofer. 2024. Exploring the Potential of Large Lan- guage Models for Automation in Technical Customer Service.arXiv preprint arXiv:2405.09161(2024). https://arxiv.org/abs/2405.09161

    2024 arXiv

  14. [22]

    Xin Zhang, Tianjie Ju, Huijia Liang, Ying Fu, and Qin Zhang. 2024. LLMs In- struct LLMs:An Extraction and Editing Method. doi:10.48550/arXiv.2403.15736 arXiv:2403.15736 [cs] version: 1. From Unstructured Communication to Intelligent RAG: Multi-Agent Automation for Supply Chain...

    2024 doi arXiv

  15. [24]

    Provide a clear , concise name to capture the essence of the issue type (5 words or less )

  16. [25]

    Write a brief description of what this category encompasses (50 words or less )

  17. [26]

    List identifying patterns or keywords ( maximum 15 per category ) # Important Guidelines

  18. [27]

    Focus ONLY on categories that are ACTUALLY REPRESENTED in the sample tickets

  19. [28]

    Categories should be based on the nature of the problem , not just surface details

  20. [29]

    Categories should be distinct from each other with minimal overlap

  21. [30]

    The number of categories should reflect the diversity in the sample - DO NOT create more categories than justified by the samples

  22. [31]

    Be extremely concise with category names and description , and use short keywords for identifying patterns

  23. [32]

    DO NOT create categories for general organizational content that isn't an actual problem

  24. [33]

    categories

    DO NOT use your general knowledge about supply chain systems to invent categories - rely ONLY on what's in the data # Output Format Return a JSON structure : { " categories ": [ { " name ": " Short Category Name " , " description ": " Brief description " , " i d e n t i f y i ...

  25. [34]

    Identical categories that appear in multiple sets

  26. [35]

    Similar categories with slight variations in name , description , or identifying patterns

  27. [36]

    Unique categories that only appear in one set For the merged taxonomy :

  28. [37]

    Consolidate identical or highly similar categories

  29. [38]

    For similar categories , combine the best elements of each description and merge identifying patterns

  30. [39]

    Preserve unique categories that represent distinct knowledge areas

  31. [40]

    Keep descriptions brief (50 words maximum )

  32. [41]

    Limit to maximum 15 identifying patterns per category # Important Guidelines

  33. [42]

    Zhang et al

    Focus on semantic similarity , not just text matching KDD ’25, August 3, 2025, Toronto, ON, Canada. Zhang et al. Table 2: Comparison of recent LLM/RAG/agentic systems for extracting structured knowledge from enterprise data (LLM = large language model; RAG = retrieval-augmente...

    2025

  34. [43]

    When merging similar categories , choose the clearest name and most comprehensive description

  35. [44]

    Combine identifying patterns from similar categories but avoid redundancy

  36. [45]

    Ensure the final taxonomy is comprehensive with minimal redundancy

  37. [46]

    categories

    Be extremely concise with category names and descriptions # Output Format Return a JSON structure with the merged categories : { " categories ": [ { " name ": " Short Category Name " , " description ": " Brief description " , " i d e n t i f y i n g _ p a t t e r n s ": [" pat...

    2025

  38. [47]

    Provide a clear , concise name to capture the specific issue type (5 words or less )

  39. [48]

    Write a brief description of what this subcategory encompasses (50 words or less )

  40. [49]

    List identifying patterns or keywords ( maximum 10 per subcategory ) # Important Guidelines

  41. [50]

    Focus ONLY on subcategories that are ACTUALLY REPRESENTED in the sample tickets

  42. [51]

    The subcategories should be distinct from each other with minimal overlap

  43. [52]

    The number of subcategories should reflect the diversity in the sample - DO NOT create more subcategories than justified by the samples

  44. [53]

    The subcategory names should clearly relate to the parent category but be more specific

  45. [54]

    Be extremely concise with subcategory names and descriptions

  46. [55]

    subcategories

    DO NOT use your general knowledge to invent subcategories - rely ONLY on what's in the data # Output Format Return a JSON structure : { " subcategories ": [ { " name ": " Short Subcategory Name " , " description ": " Brief description " , " i d e n t i f y i n g _ p a t t e r ...

    2025

  47. [56]

    Use ONLY information explicitly mentioned in the ticket data

  48. [57]

    DO NOT expand acronyms unless they are expanded in the tickets themselves

  49. [58]

    DO NOT make up definitions for systems if not provided in the data

  50. [59]

    DO NOT invent processes or best practices not mentioned in tickets

  51. [60]

    Keep the article SHORT and FOCUSED - aim for 50% less content than you might typically write

  52. [61]

    Write in a direct style addressing ticket creators Focus on :

  53. [62]

    Common issues seen in these tickets ( briefly )

  54. [63]

    Actual solutions that worked ( from ticket resolutions )

  55. [64]

    Minimal , specific advice based only on ticket content # Output Format Your response should be a concise markdown document with :

  56. [65]

    Title : A brief descriptive title

  57. [66]

    Common Issues : 2 -3 bullet points of the main issues ( be brief )

  58. [67]

    Tips for Resolution : Specific advice based ONLY on what worked in the tickets

  59. [68]

    Resources : Only mention systems / links that appear in the tickets Total length should be no more than 400 -500 words maximum . B.7 Knowledge Merge Prompt This prompt is used for merging multiple knowledge articles on the same topic: You are merging multiple knowledge article...

  60. [69]

    ORGANIZE information logically into sections ( Common Issues , Tips for Resolution , Resources )

  61. [70]

    REMOVE redundancy - multiple articles may cover the same points

  62. [71]

    PRIORITIZE information that appears in multiple articles

  63. [72]

    INCLUDE unique insights from individual articles if they add value

  64. [73]

    MAINTAIN conciseness - focus on the most valuable information

  65. [74]

    USE the same level of specificity as the input articles

  66. [75]

    DO NOT introduce new information not present in the source articles

  67. [76]

    DO NOT expand acronyms unless they were expanded in the source articles # Output Format Your response should be a single markdown document with :

  68. [77]

    Title : A clear descriptive title related to the category

  69. [78]

    Common Issues : Consolidated list of key issues ( brief bullet points )

  70. [79]

    Tips for Resolution : Specific advice based on the source articles

  71. [80]

    From Unstructured Communication to Intelligent RAG: Multi-Agent Automation for Supply Chain Knowledge Bases KDD ’25, August 3, 2025, Toronto, ON, Canada

    Resources : Systems / links that appear in the source articles Total length should be no more than 400 -500 words maximum . From Unstructured Communication to Intelligent RAG: Multi-Agent Automation for Supply Chain Knowledge Bases KDD ’25, August 3, 2025, Toronto, ON, Canada....

    2025

  72. [81]

    Accuracy : Does the answer provide factually correct information based on the ticket ?

  73. [82]

    Completeness : Does the answer address all aspects of the question ?

  74. [83]

    Relevance : Does the answer focus on what was actually asked ?

  75. [84]

    Clarity : Is the answer easy to understand ? # Output Format Provide your evaluation in the following format :

  76. [85]

    Helpfulness Score (1 -5 , where 1 is not helpful at all and 5 is extremely helpful )

  77. [86]

    Reasoning : Brief explanation for your score

  78. [87]

    Missing Information : Any critical information from the ticket that should have been included

  79. [88]

    This comprehensive evaluation framework ensured a rigorous and consistent assess- ment of RAG system performance across different knowledge base creation approaches

    Improvement Suggestions : How the answer could be made more helpful The evaluation process assessed each answer on a 5-point help- fulness scale: •1: Not helpful at all •2: Slightly helpful •3: Moderately helpful •4: Very helpful •5: Extremely helpful The evaluation focused on...

  80. [89]

    Be phrased as a clear , specific question or problem statement

  81. [90]

    Include relevant context ( site codes , order numbers , etc .) if they are critical to understanding the issue

  82. [91]

    Focus on what the requester needs help with

  83. [92]

    Be concise but complete ( typically 5 -15 words ) # Output Format Return ONLY the query text with no prefixes , explanations , or surrounding quotes . This process transformed ticket information into natural lan- guage queries that simulate how users would typically ask for as...

  84. [2025]

    doi:10.48550/arXiv.2504.03165 arXiv:2504.03165 [cs]

    Efficient Dynamic Clustering-Based Document Compression for Retrieval- Augmented-Generation. doi:10.48550/arXiv.2504.03165 arXiv:2504.03165 [cs]

    doi arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.