Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

LLM as HPC Expert: Extending RAG Architecture for HPC Data

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that an HPC question-answering system answers users more accurately when, instead of retrieving only documentation chunks, it also matches the query to a natural-language description of a shell command, executes that…

desk verdict Plausible HyDE-style extension to shell-command retrieval for HPC Q&A, with believable qualitative gains but a self-referential LLM benchmark behind the headline number; worth a serious referee. read the letter →

arxiv 2501.14733 v1 pith:BM7B6KJS submitted 2024-12-09 cs.DC cs.AI

Yusuke Miyashita , Patrick Kin Man Tung , Johan Barthélemy This is my paper · ORCID
classification cs.DCcs.AI
keywords retrieval-augmentedgenerationhypotheticalcommandembeddingshigh-performancecomputingLLM-as-a-judgesyntheticquestion-answerevaluationshellretrievalexecutionsafetyRAG
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes Hypothetical Command Embeddings (HyCE), an extension of retrieval-augmented generation (RAG) for high-performance computing support. The idea is that many HPC questions, such as which GPUs are available to the current user or what the status of a submitted job is, cannot be answered from static documentation because the answer lives in the user's live session. HyCE adds a retrieval step in which the user query is matched against plain-language descriptions of shell commands; the top command is then executed, and its output is included in the context the LLM uses to answer. The paper reports that this raises an LLM-judged evaluation score from 77.67% to 82.33%, and that further prompt and model improvements bring the combined pipeline to 86%. A sympathetic reader would care because it offers a path for LLMs to act on real-time, user-specific system data without fine-tuning and with safeguards around command execution.

What carries the argument

The machinery is the HyCE retrieval step. Each shell command in the corpus is paired with a short, natural-language description of what it does. Given a user query, a cross-encoder scores the query against these descriptions rather than against the raw command strings; the top command is retrieved, executed in the user's environment, and its output is added to the document context. This borrows the hypothetical-document-embedding idea of embedding a hypothetical version of what you want to retrieve, but applies it to commands via their descriptions. Supporting machinery is the automatic evaluation loop: an LLM writes synthetic question-answer pairs from the chunks, filters them, generates answers through the RAG pipeline, and judges those answers against the reference pairs using binary correctness and faithfulness scores.

What would settle it

A reader could test the central claim by taking the released 100-question set, running the pipeline with human experts scoring the same answers, and checking whether human-assigned correctness and faithfulness scores reproduce the 4.66-percentage-point improvement from HyCE. A second check is to measure retrieval precision directly, comparing how often HyCE retrieves the intended command for a query such as a GPU-availability question versus direct command-query matching.

Watch

Extended reading notes

Core claim

The central claim is that matching a user's natural-language query to a description of a shell command, then executing the retrieved command and appending its output to the retrieval context, produces more correct and more faithful HPC answers than document-only RAG. Commands are abbreviated and syntactically structured, so direct embedding of command text matches user queries poorly; descriptions such as "this command checks the GPU model, memory usage, and utilization rate in real time" align with natural language much better. With the command output in context, the LLM can answer questions that are unanswerable from documentation alone, such as which GPUs the current user can access. The paper reports this as a 4.66 percentage-point gain in an automated evaluation where the LLM generates 100 synthetic HPC questions from the data chunks, filters them for groundedness, relevance, and standalone quality, answers them through the RAG pipeline, and judges correctness and faithfulness against its own reference answers.

Load-bearing premise

The evaluation assumes that the same LLM that builds the benchmark and writes the reference answers also judges whether the system's answers are correct and faithful; if that judge prefers its own style of answer, the measured 4.66% gain may not reflect real-user benefit.

Editorial extensions

If this is right

  • HPC support chatbots can answer user-specific questions like GPU availability and job status that static documentation alone cannot answer.
  • HyCE composes with other RAG improvements: chain-of-thought prompting adds 0.67% and better retrieval, re-rank, and LLM choices add 3%, suggesting the gains are additive.
  • The automatic evaluation framework can be run continuously on a cluster's own documentation, giving a scalable way to track RAG quality as documentation and commands evolve.
  • Safety layers, including a predefined command whitelist, user-level privileges, and containerization, make LLM-driven command execution practical to deploy in shared HPC environments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 100-question benchmark contains only 10 command-derived questions, so the reported 4.66% HyCE gain rests on a small command sample; a benchmark weighted toward command-centric questions would likely show a larger or more variable effect.
  • Because HyCE executes commands on the user's behalf, its usefulness depends on a maintained corpus of command descriptions; organizations would need a process to keep descriptions in sync as new commands and cluster policies appear.
  • The description-matching trick could generalize beyond HPC to any domain where the answer to a user query lives in the output of a tool, such as database queries, cloud CLIs, or git commands, not just in documents.
  • A natural next experiment is human evaluation: having real HPC users rate answers from document-only RAG versus HyCE, which would directly test whether the LLM judge's preferences align with user satisfaction.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes HyCE, an extension of retrieval-augmented generation (RAG) for HPC support, in which a user query is matched against natural-language descriptions of predefined shell commands; the retrieved command is executed and its output is added to the LLM context. The authors also propose an automatic evaluation framework in which the LLM generates synthetic Q&A pairs from HPC documentation and command chunks, filters them, answers via the RAG pipeline, and then scores correctness and faithfulness against the synthetic references. Experiments on the Katana cluster (Section 5, Table 2) report that HyCE raises the evaluation score from 77.67% to 82.33%, with further gains from chain-of-thought prompting and better retrieval/rerank/LLM models, reaching 86% overall. The code and prompts are open-sourced, and the paper includes a discussion of security considerations.

Significance. If the quantitative claims hold, HyCE provides a simple and practical way to give RAG-based HPC assistants access to real-time, user-specific information, and the idea of retrieving commands through natural-language descriptions rather than raw command strings is sensible and likely useful. The release of open-source code and prompt templates is a concrete contribution. However, the current evidence is not convincing: the evaluation is self-referential (the same model family generates the benchmark, answers, and judges), the quantitative results are single point estimates from one run over 100 synthetic pairs, and only 10 pairs involve commands, the very part that HyCE is designed to improve. The qualitative example in Figure 2 illustrates the intended mechanism but is anecdotal. The paper's central claim is defensible and worth pursuing, but it requires substantially stronger validation.

major comments (3)
  1. [Section 5.2, Table 2, Algorithm 1] The reported 4.66 percentage-point gain from HyCE rests entirely on an automatic evaluation loop in which the same LLM family creates the 100 synthetic Q&A pairs from the chunks (Section 4.1), filters them by groundedness/relevance/standalone criteria, generates answers through the RAG pipeline, and then judges correctness and faithfulness against those synthetic references. The Correctness criterion compares the generated answer with the synthetic reference answer rather than with an independent ground truth, which can reward answers that simply match the generator's own style. No human spot-checks, no inter-annotator agreement, no error bars or statistical tests, and no separate reporting for the 10 command-based pairs (which are the only ones HyCE can affect) are provided. Because only 10 of 100 questions exercise the HyCE mechanism, a handful of judgment changes would alter the claimed conclusion. The authors should add a human evaluation on a representative sample, report confidence intervals or variance across multiple runs, and break down scores by question type (command-based vs documentation-based).
  2. [Section 3.3, Table 1] Table 1 compares average top similarity scores across five different cross-encoders, but the raw scores from different models are on different scales, so cross-model comparisons are meaningless. The only valid comparison is within each row (query vs command name, query vs command description), and even there no variance or significance is reported. Moreover, this table measures only a proxy of retrieval quality; the actual end-to-end effect is what Table 2 purportedly measures. Given the small differences in the first row (-2.8283 vs -2.7880), the claim that HyCE 'consistently' improves matching should be backed by a statistical test or at least by reporting per-example distributions.
  3. [Section 2.1 and Section 5] The paper positions HyCE against existing command-generation tools such as ShellGPT and AI-Shell, but the evaluation does not include any comparison with these or with a simple RAG baseline that retrieves command documentation text without execution. Without such a baseline, it is unclear whether the observed gain comes from HyCE's command retrieval and execution specifically, from the richer context, or from the synthetic evaluation procedure itself. A comparison against an existing tool (or a plausible non-execution baseline) would substantially strengthen the central claim.
minor comments (5)
  1. [Section 1] The sentence 'allowing them to serve as HPC experts HPC users' appears to be missing a word; it should likely read 'serve as HPC experts for HPC users'.
  2. [Section 3.2] The phrase 'HPC data encompasses to cluster documentations and shell commands' contains a small grammatical error; 'to' should be removed, and 'documentations' is normally uncountable ('documentation').
  3. [Section 5.2] The phrase 'in table 2' is inconsistently capitalized; elsewhere the paper uses 'Table 2', and the reference in the text should match.
  4. [Section 4.1] The description of Q&A generation for command chunks states that 'commands are executed, and both the command description and output are used in LLM to create the Q&A pairs.' This means the reference answers are in part derived from the command output, which is a source of information leakage in the self-evaluation loop; this should be stated explicitly as a limitation in Section 5.3.
  5. [Appendix B] The prompt templates are useful, but the paper says 'Please refer to our GitHub repository for full details'; since the repository is open-source, it would help to include the exact version or commit hash in the manuscript for reproducibility.

Circularity Check

1 steps flagged · score 6.0 of 10

The 4.66% HyCE gain is measured on a self-referential LLM-as-judge benchmark: the same model family writes the questions, writes the reference answers, and scores the RAG outputs against those references.

  1. self definitional [Abstract, Section 4.1, Algorithm 1, Section 4.2, Figure 6]
    "We evaluate HyCE using an automated RAG evaluation framework, where the LLM itself creates synthetic questions from the HPC data and serves as a judge. ... (𝑄ℎ𝑦𝑝,𝐴ℎ𝑦𝑝)← LLM_Synthetic_Data_Gen(D) ... Eval_Score← LLM_Eval(𝐴′ℎ𝑦𝑝,𝐴𝑝𝑟𝑒𝑑,D)"

    The benchmark is not an independent test set. In Algorithm 1, the same LLM family generates the synthetic questions and reference answers from the same chunks that RAG will retrieve, then filters them, then acts as the judge. The judge's Correctness criterion (Figure 6) compares the generated answer to the reference answer rather than to any external ground truth. Thus the reported 77.67% baseline and 82.33% HyCE score measure how well the RAG pipeline reproduces the generator's own style and content. The 4.66-point gain attributed to HyCE is therefore computed on a self-authored, self-scored exam; it is not anchored to human judgments or an independent command-generation benchmark.

full rationale

The paper's central quantitative claim—that HyCE raises RAG accuracy from 77.67% to 82.33% (Table 2, Section 5.2)—rests entirely on the automatic evaluation pipeline in which the LLM generates the synthetic Q&A pairs, filters them, answers them through RAG, and scores correctness and faithfulness against its own generated references (Algorithm 1, Section 4.1-4.2). This is a genuine self-referential evaluation design: the exam writer and the examiner are the same model family being evaluated, and the reference answers are produced from the same chunks used for retrieval. The paper itself acknowledges in Section 5.3 that the evaluation cannot measure performance outside the provided chunks, but it does not address the additional fragility that the judge compares to LLM-generated references rather than to verified ground truth, nor does it provide human spot-checks, error bars, or comparison with existing command-generation baselines. Some independent evidence does exist: Table 1 directly measures cross-encoder similarity between queries and command descriptions versus raw commands, and Figure 2 shows a plausible qualitative benefit from HyCE. There are no load-bearing self-citations; the cited prior work is external. Because part of the evidence is independent but the headline quantitative result is defined by the LLM's own judgment, the appropriate finding is partial circularity rather than full circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new physical or conceptual entities are introduced. The free parameters are hyperparameters and the composition of the synthetic evaluation set. The axioms reflect the main domain assumptions: synthetic data represent real users, the LLM judge is reliable, command execution is safe and correct, and semantic similarity is a valid retrieval proxy. The invented-entities list is empty because HyCE is a retrieval procedure, not a postulated object.

free parameters (4)
  • Retrieval top-K = 20
    Hand-selected in Appendix A. No sensitivity analysis is reported, yet this value controls how many command descriptions and document chunks reach the reranker.
  • Re-rank top-K = 5
    Hand-selected in Appendix A. This controls the final context size and therefore directly influences answer quality.
  • Synthetic Q&A composition = 90 documentation + 10 command pairs
    Arbitrary split in Section 4.1. With only 10 command-based pairs, the HyCE gain in Table 2 is estimated on a very small command sample.
  • Chunk size = Determined by LLM
    Appendix A says chunk size is determined by the LLM rather than fixed. This makes the retrieval index depend on the generating model's choices.
assumptions (4)
  • domain assumption LLM-generated synthetic questions and reference answers from the chunks are representative of real HPC user queries.
    Section 4.1 uses the LLM itself to create and then filter Q&A pairs; whether these resemble real user questions is assumed, not tested against human users.
  • domain assumption An LLM judge's binary correctness and faithfulness scores are a valid measure of answer quality.
    Section 4.2 delegates scoring to the same model family without calibration against human labels or known benchmarks; prior work on LLM-as-judge is cited but not used to validate this setup.
  • domain assumption Executing the top retrieved predefined command on the user's behalf is safe and produces output that answers the question.
    Section 3.3 assumes the retrieved command is the right one, and Section 6.2 assumes predefined commands plus user-level privileges contain risk. No false-execution or safety experiment is reported.
  • domain assumption Semantic similarity between a query and a command description is a valid proxy for retrieval usefulness.
    Table 1 reports higher similarity scores with descriptions, but no downstream accuracy test links these similarity gains to answer correctness.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLM as HPC Expert: Extending RAG Architecture for HPC Data." pith.science (2026). https://pith.science/paper/BM7B6KJS

@misc{pith2026250114733,
  author       = {Pith},
  title        = {Pith review of: LLM as HPC Expert: Extending RAG Architecture for HPC Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BM7B6KJS}},
  note         = {Machine review of arXiv:2501.14733}
}
read the original abstract

High-Performance Computing (HPC) is crucial for performing advanced computational tasks, yet their complexity often challenges users, particularly those unfamiliar with HPC-specific commands and workflows. This paper introduces Hypothetical Command Embeddings (HyCE), a novel method that extends Retrieval-Augmented Generation (RAG) by integrating real-time, user-specific HPC data, enhancing accessibility to these systems. HyCE enriches large language models (LLM) with real-time, user-specific HPC information, addressing the limitations of fine-tuned models on such data. We evaluate HyCE using an automated RAG evaluation framework, where the LLM itself creates synthetic questions from the HPC data and serves as a judge, assessing the efficacy of the extended RAG with the evaluation metrics relevant for HPC tasks. Additionally, we tackle essential security concerns, including data privacy and command execution risks, associated with deploying LLMs in HPC environments. This solution provides a scalable and adaptable approach for HPC clusters to leverage LLMs as HPC expert, bridging the gap between users and the complex systems of HPC.

Figures

Figures reproduced from arXiv: 2501.14733 by the authors.

Figure 1
Figure 1. RAG Architecture with HPC Data. In addition to the conventional RAG architecture, it incorporates Hypothetical [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Example of comparing of two RAG responses. Unlike the ambiguous response provided by RAG with just HPC [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The prompts for RAG Q&A generation system_message = "You are a helpful assistant that evaluates questions." user_message = f""" You will be given a context and a question. Your task is to evaluate the question based on the following criteria: 1. **Groundedness**: The question can be answered unambiguously with the given context. 2. **Relevance**: The question addresses a common, practical problem or concept that HPC… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The prompts for RAG Q&A filtering [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: The prompts for RAG answering system_message = "You are a helpful assistant that evaluates answers." user_message = f""" You will be given a question, a generated answer, and a reference answer. Your task is to evaluate the generated answer based on the following crite…
Figure 6
Figure 6. Figure 6: The prompts for RAG evaluation [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. HARGO: Heterogeneity-Aware Reward-Guided Optimization for RL Post-Training of LLMs on HPC Tasks

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Confidence-modulated per-response advantage weighting (HARGO) improves GRPO-style RL post-training on four heterogeneous HPC tasks, leading WinRate, data-race F1, and PLP similarity at 0.5B.

Reference graph

Works this paper leans on

26 extracted references · 4 canonical work pages · cited by 1 Pith paper

  1. [1]

    Vaibhav Adlakha, Parishad BehnamGhader, Xing Han Lu, Nicholas Meade, and Siva Reddy. 2024. Evaluating Correctness and Faithfulness of Instruction- Following Models for Question Answering. arXiv:2307.16877 [cs.CL] https: //arxiv.org/abs/2307.16877

    arXiv 2024

  2. [2]

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. arXiv:2310.11511 [cs.CL] https://arxiv.org/abs/2310.11511

    arXiv 2023

  3. [3]

    Builder.io. 2023. AI Shell. https://github.com/BuilderIO/ai-shell/tree/main. Ac- cessed: 2024-11-24

    work page 2023

  4. [4]

    Ahmed, Akash Dutta, Arijit Bhattacharjee, Sixing Yu, Quazi Ishtiaque Mahmud, Waqwoya Abebe, Hung Phan, Aishwarya Sarkar, Branden Butler, Niranjan Hasabnis, Gal Oren, Vy A

    Le Chen, Nesreen K. Ahmed, Akash Dutta, Arijit Bhattacharjee, Sixing Yu, Quazi Ishtiaque Mahmud, Waqwoya Abebe, Hung Phan, Aishwarya Sarkar, Branden Butler, Niranjan Hasabnis, Gal Oren, Vy A. Vo, Juan Pablo Munoz, Theodore L. Willke, Tim Mattson, and Ali Jannesari. 2024. The Landscape and Challenges of HPC Research and LLMs. arXiv:2402.02018 [cs.LG] https...

    arXiv 2024

  5. [5]

    Le Chen, Pei-Hung Lin, Tristan Vanderbruggen, Chunhua Liao, Murali Emani, and Bronis de Supinski. 2023. LM4HPC: Towards Effective Language Model Ap- plication in High-Performance Computing . Springer Nature Switzerland, 18–33. https://doi.org/10.1007/978-3-031-40744-4_2

    doi:10.1007/978-3-031-40744-4_2 2023

  6. [6]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 [cs.CL] https://arxiv.org/abs/1810.04805

    arXiv 2019

  7. [7]

    Xianzhong Ding, Le Chen, Murali Emani, Chunhua Liao, Pei-Hung Lin, Tris- tan Vanderbruggen, Zhen Xie, Alberto Cerpa, and Wan Du. 2023. HPC-GPT: Integrating Large Language Model for High-Performance Computing. In Pro- ceedings of the SC ’23 Workshops of The International Conference on High Perfor- mance Computing, Network, Storage, and Analysis (SC-W 2023)...

    arXiv 2023

  8. [8]

    Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert

Show all 26 references
  1. [9]

    Abhimanyu Dubey et al. 2024. The Llama 3 Herd of Models. arXiv:2407.21783 [cs.AI] https://arxiv.org/abs/2407.21783

    2024 arXiv

  2. [10]

    Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2022. Precise Zero-Shot Dense Retrieval without Relevance Labels. arXiv:2212.10496 [cs.IR] https: //arxiv.org/abs/2212.10496

    2022 arXiv

  3. [11]

    Michael Günther, Isabelle Mohr, Daniel James Williams, Bo Wang, and Han Xiao. 2024. Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models. arXiv:2409.04701 [cs.CL] https://arxiv.org/abs/2409.04701

    2024 arXiv

  4. [12]

    Hudak, Douglas Johnson, Jeremy Nicklas, Eric Franz, Brian McMichael, and Basil Gohar

    David E. Hudak, Douglas Johnson, Jeremy Nicklas, Eric Franz, Brian McMichael, and Basil Gohar. 2016. Open OnDemand: Transforming Computational Science Through Omnidisciplinary Software Cyberinfrastructure. In Proceedings of the XSEDE16 Conference on Diversity, Big Data, and Sc...

    2016

  5. [13]

    Vo, Nadav Schneider, Neva Krien, Mihai Capota, Abdul Wasay, Nesreen Ahmed, Ted Willke, Guy Tamir, Yuval Pinter, Timothy Mattson, and Gal Oren

    Tal Kadosh, Niranjan Hasabnis, Vy A. Vo, Nadav Schneider, Neva Krien, Mihai Capota, Abdul Wasay, Nesreen Ahmed, Ted Willke, Guy Tamir, Yuval Pinter, Timothy Mattson, and Gal Oren. 2024. MonoCoder: Domain-Specific Code Language Model for HPC Codes and Tasks. arXiv:2312.13322 [c...

    2024 arXiv

  6. [14]

    Vo, Nadav Schneider, Neva Krien, Abdul Wasay, Nesreen Ahmed, Ted Willke, Guy Tamir, Yuval Pinter, Timothy Mattson, and Gal Oren

    Tal Kadosh, Niranjan Hasabnis, Vy A. Vo, Nadav Schneider, Neva Krien, Abdul Wasay, Nesreen Ahmed, Ted Willke, Guy Tamir, Yuval Pinter, Timothy Mattson, and Gal Oren. 2023. Scope is all you need: Transforming LLMs for HPC Code. arXiv:2308.09440 [cs.CL] https://arxiv.org/abs/2308.09440

    2023 arXiv

  7. [15]

    Omar Khattab and Matei Zaharia. 2020. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. arXiv:2004.12832 [cs.IR] https://arxiv.org/abs/2004.12832

    2020 arXiv

  8. [16]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2021. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv:2005.1140...

    2021 arXiv

  9. [17]

    Daniel Nichols, Aniruddha Marathe, Harshitha Menon, Todd Gamblin, and Ab- hinav Bhatele. 2024. HPC-Coder: Modeling Parallel Programs using Large Lan- guage Models. In ISC High Performance 2024 Research Paper Proceedings (39th International Conference). IEEE, 1–12. https://doi....

    2024

  10. [18]

    Zackary Rackauckas. 2024. RAG-Fusion: A New Take on Retrieval Augmented Generation. International Journal on Natural Language Computing 13, 1 (Feb. 2024), 37–47. https://doi.org/10.5121/ijnlc.2024.13103

    2024

  11. [19]

    Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. arXiv:1908.10084 [cs.CL] https://arxiv.org/abs/ 1908.10084

    2019 arXiv

  12. [20]

    TheR1D. 2024. Shell GPT. https://github.com/TheR1D/shell_gpt. Accessed: 2024-11-24

    2024

  13. [21]

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal

  14. [22]

    University of New South Wales. 2014. Katana. https://doi.org/10.26190/669x-a286. Accessed: 2024-11-24

    2014 doi

  15. [23]

    arXiv:2212.10509 [cs.CL] https://arxiv.org/abs/ 2212.10509

    Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge- Intensive Multi-Step Questions. arXiv:2212.10509 [cs.CL] https://arxiv.org/abs/ 2212.10509

    arXiv

  16. [24]

    Youareahelpfulassistantthatgeneratesfactoidquestionsandanswers

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv:2306.05685 [cs.CL] https://ar...

    2023 arXiv

  17. [25]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv:2201.11903 [cs.CL] https: //arxiv.org/abs/2201.11903

    2023 arXiv

  18. [2023]

    arXiv:2309.15217 [cs.CL] https://arxiv.org/abs/2309.15217

    RAGAS: Automated Evaluation of Retrieval Augmented Generation. arXiv:2309.15217 [cs.CL] https://arxiv.org/abs/2309.15217

    arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.