Pith. sign in

REVIEW 3 major objections 6 minor 41 references

Themes of Building LLM-based Applications for Production: A Practitioner's View

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Practitioner talks on production LLM work cluster into 20 topics across 8 themes, with retrieval-augmented generation the most prevalent.

desk verdict A transparent, useful thematic map of what practitioners discuss when building LLM apps, but the prevalence numbers are directional because the corpus selection and classification have unmeasured biases. read the letter →

arxiv 2411.08574 v2 pith:ZBQCHPMF submitted 2024-11-13 cs.SE

Alina Mailach , Sebastian Simon , Johannes Dorn , Norbert Siegmund This is my paper · ORCID
classification cs.SE
keywords LLMinproductionretrieval-augmentedgenerationRAGsystemstopicmodelingBERTopicpractitionerexperiencethematicanalysisLLMOps
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to give a comprehensive overview of what practitioners actually discuss when building and deploying large language model (LLM) applications, a topic the authors argue existing software-engineering research has not yet mapped. It claims that these discussions cluster into 20 topics within 8 themes, and that the single most prevalent topic is retrieval-augmented generation (RAG), appearing in 40.4% of the 189 analyzed practitioner videos. A sympathetic reader should care because the resulting thematic map gives practitioners a checklist of concerns — architecture, model techniques, infrastructure, evaluation, risks, monitoring, costs, output verification — and gives researchers a list of industrially relevant problems. The analysis is semi-automated: transcripts are clustered with BERTopic, then manually reviewed, merged, and grouped into themes, yielding a map that can guide both project planning and future research.

What carries the argument

The machinery is a semi-automated topic-analysis pipeline. Transcripts of 189 videos (over 92 hours) are produced with the whisper-large speech recognizer, split by a semantic text splitter into about 4100 coherent paragraphs, and clustered with BERTopic, a neural topic model that groups paragraphs by semantic similarity and labels them with shared vocabulary. Two researchers then manually inspect the generated topics, remove incoherent or irrelevant clusters, merge related ones, and group the survivors into eight themes; a video is counted as discussing a topic if any of its paragraphs fall in that topic's cluster. This combination matters because the automated step makes the 92-hour corpus tractable, while the manual step is what turns raw clusters into the 20 interpretable topics and the co-occurrence matrix that carries the paper's claims.

What would settle it

Compile an independent, broader sample of practitioner discussions — for instance, all English-language talks from major AI engineering conferences in 2023–2024 plus a large sample of technical blog posts — and count topic prevalence with a pre-registered codebook. If retrieval-augmented generation is not the most frequent topic, or if Architecture & Design is not the largest theme, then the paper's prevalence claims and theme ranking are artifacts of its video-selection filter rather than a true map of practitioner priorities.

Watch

Extended reading notes

Core claim

The central claim is that the concerns of practitioners building LLM-based systems in production are not scattered but fall into a stable structure: eight themes — Architecture & Design, Model Capabilities & Techniques, Tools, Infrastructure & LLM Providers, Evaluation, Risks & Ethics, Monitoring & Observability, Costs, and Output Verification — containing twenty topics. The paper reports that Architecture & Design is the largest theme (52.2% of videos) and that RAG systems are the most frequently discussed single topic (72 videos, 40.4%), followed by fine-tuning (24.2%), prompt engineering (23.0%), and application security (18.5%). It also reports co-occurrences among topics, showing, for example, that evaluation is the most common companion of RAG discussions and that RAG, fine-tuning, prompt engineering, and agents are often discussed as alternative techniques for the same problem. In the paper's own framing, this thematic map is the contribution: it offers practitioners a starting point for what to consider and researchers a set of open, practice-driven topics such as systematic evaluation of RAG, architectural guidance for agents, and production-oriented prompt engineering.

Load-bearing premise

The whole map rests on the assumption that the 189 videos selected from five YouTube queries, after manually excluding tutorials, lectures, tool introductions, and non-English content, fairly represent what practitioners consider important when building and deploying LLM applications.

Editorial extensions

If this is right

  • If the map is right, RAG engineering — chunking, embeddings, retrieval optimization, reranking — is the dominant practical bottleneck in production LLM work, ahead of model training and tuning.
  • Evaluation is a cross-cutting concern: it is the most frequent co-occurring topic with RAG, and it co-occurs heavily with fine-tuning and prompt engineering, so any production workflow needs an evaluation strategy tied to the chosen technique.
  • Fine-tuning and RAG appear together in 19 videos largely because practitioners treat them as alternative routes to the same goal (grounding a model in private or recent data), implying that architecture choices are often framed as technique trade-offs rather than additive stack components.
  • Cost discussions concentrate on context length and token-based pricing, and output-verification discussions center on guardrails; both are themes that the paper says receive little software-engineering research, so they mark concrete gaps for future work.
  • The co-occurrence matrix lets a practitioner starting with one topic (say, RAG) see which neighboring concerns to plan for, such as latency, application security, memory, and user interfaces.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the corpus deliberately excludes tutorials, lectures, and tool introductions, the map reflects experience reports and conference talks; hands-on how-to concerns could be more prominent in the broader practitioner population than these prevalence numbers show.
  • Editorial inference: 1843 of 4100 paragraphs (about 45%) were left unclassified by BERTopic and treated as transitions; a different topic model or a lower classification threshold might surface niche topics that the current 20-topic map misses, especially around operations and incident response.
  • Editorial inference: the five query strings and English-language filter bound the map; a replication on non-English content or on written practitioner spaces (technical blogs, Q&A forums) could shift the theme ranking and is a direct test of generality.
  • Editorial inference: the co-occurrence matrix could plausibly be turned into a practical recommender — given a chosen architecture, list the topics that practitioners most often pair with it — which the paper presents as a future possibility rather than a built tool.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents a semi-automated thematic analysis of 189 YouTube videos (2022–2024) in which practitioners discuss building and deploying LLM-based applications. The authors transcribe the videos, split the transcripts into 4100 paragraphs, apply BERTopic, and then manually assess, merge, and group the resulting topics into 20 topics across 8 themes. The main contribution is a thematic map with prevalence counts and a co-occurrence matrix, and the headline findings are that RAG Systems is the most frequently discussed topic (72 videos, reported as 40.4%) and that Architecture & Design is the largest theme (93 videos, reported as 52.2%). The paper also discusses implications for practitioners and research directions.

Significance. If the prevalence claims are reliable, the thematic map is a useful resource for both practitioners and software engineering researchers: it consolidates a scattered body of practitioner discussions into a structured set of topics and themes, and it identifies RAG as the dominant concern. The paper provides a replication package with a mapping of videos to topics, which is a concrete strength. However, the quantitative prevalence claims are central to the paper's contribution, and they currently rest on a sampling and filtering process that is partly subjective and on a topic-model output in which 45% of paragraphs are left unclassified. The internal inconsistency in the reported percentages further weakens the quantitative claims. The qualitative thematic map itself is defensible and likely valuable, but the prevalence ranking needs additional support before the paper's central claims can be accepted as stated.

major comments (3)
  1. [Section IV-A, Section III-B] The prevalence percentages in Section IV-A are not computed on the stated sample of 189 videos. For example, 93/189 = 49.2%, not the reported 52.2%, and 72/189 = 38.1%, not 40.4%. All reported percentages are exactly consistent with a denominator of 178 videos (e.g., 93/178 = 52.2%, 72/178 = 40.4%). The methodology does not mention excluding 11 videos or using a different denominator. This is load-bearing because the paper's headline result is a prevalence ranking. The authors must state the exact denominator used for each percentage, explain any exclusion of videos (e.g., videos with no classified paragraphs), and recompute or justify all prevalence figures.
  2. [Section III-B] The paper states that 1843 of 4100 paragraphs (45%) remained unclassified by BERTopic and asserts that this is expected because such paragraphs are transitions, introductions, or greetings that carry no distinguishing topic information. This assertion is not supported by evidence, and it is consequential: if a substantial share of the unclassified paragraphs are substantive, then all topic counts and co-occurrence values change. I recommend a robustness analysis, for example: (1) manually label a random sample of unclassified paragraphs and report the share that are truly non-topical, (2) re-run the topic model with a lower probability threshold or a fallback assignment that maps unclassified paragraphs to their nearest topic, or (3) at minimum report the sensitivity of the top topic rankings to plausible reassignment of unclassified paragraphs.
  3. [Section III-A] The manual relevance filter is a load-bearing step: the final corpus of 189 videos determines every prevalence estimate. Two researchers assessed videos from titles, descriptions, and occasionally the videos themselves, but the paper reports no inter-rater reliability, no disagreement-resolution procedure, and no audit of the exclusion decisions. Moreover, many retained videos come from vendor and community channels (e.g., LlamaIndex, Databricks, Cohere, Snorkel AI, MLOps.community), so the filter may preferentially retain content that showcases RAG and LLM tooling. This could make the reported prevalence ranking reflect the supply of vendor talks rather than practitioner priorities. I recommend that the authors report an inter-rater agreement measure on a sample of the relevance decisions and provide a sensitivity analysis that recomputes the theme and topic rankings after excluding videos from vendor-affiliated channels.
minor comments (6)
  1. [Abstract] The abstract contains the typo 'We further pale off topics of interest'; this should be 'highlight' (the full text uses 'highlight').
  2. [Abstract and Section IV-A] The theme is called 'Design & Architecture' in the abstract but 'Architecture & Design' in Section IV-A and elsewhere; the naming should be made consistent.
  3. [Section IV-A.2] In the first paragraph of the Model Capabilities & Techniques theme, the paper calls it 'the third largest theme', but with 83 videos it is the second largest after Architecture & Design (93 videos) and ahead of Tools, Infrastructure & LLM providers (74 videos). This ordinal claim should be corrected.
  4. [Section I] The sentence 'Analyzing these public videos has been shown to contain valuable insights that can be of comparable depth as traditional interview studies [8]' cites reference [8], which is an interview-based study on socio-technical anti-patterns and does not appear to support the claim about YouTube videos. Please provide a suitable citation or remove the unsupported claim.
  5. [Section III-B] BERTopic does not produce per-topic probabilities by default unless configured with a probability calculation option; the paper should state the exact BERTopic configuration used so that the sentence about no paragraph having more than one topic with probability above 0.2 is verifiable.
  6. [Figure 3] The figure contains typos in the axis labels: 'Managenemt' should be 'Management' and 'Co-Occurences' should be 'Co-occurrences'. The figure also has a row/column for 'Security of Cloud' while the text refers to 'Cloud Security and Privacy'; please align the terminology.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the thematic map is an empirical summary of the analyzed corpus, and no claimed prediction reduces to the paper's own inputs.

full rationale

This paper conducts a semi-automated thematic analysis of YouTube practitioner talks. The core outputs—20 topics in 8 themes and the prevalence claim that RAG Systems is the most frequently mentioned topic (72 videos, 40.4%)—are descriptive statistics computed from the collected and manually filtered corpus, not predictions derived from fitted parameters or from the paper's own definitions. The manual sorting and merging of BERTopic topics is an analytic interpretation step, but it does not make the result circular: the themes are induced from the data rather than presupposed by it. The self-citations in the paper, namely [8] (Mailach and Siegmund 2023) used as methodological motivation and [27] (Simon et al. 2024) cited as related work on RAG evaluation, are background or follow-up references and do not carry the load of the central claim. Even if the citation [8] is questionable as support for the claim about analyzing public videos, that is a correctness or validity concern, not circularity. No equation, fitted constant, uniqueness theorem, or definitional equivalence is invoked to force the reported results. The paper is self-contained as an empirical study whose claims are about the corpus it analyzed, so the circularity score is 0.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

Beyond standard tools (Whisper, BERTopic), the central claim rests on three uncontrolled assumptions: that a YouTube sample of experience reports is a valid proxy for practitioner priorities, that manual relevance filtering and topic merging are reliable, and that unclassified paragraphs do not carry topical content. The final topic count is the only hand-set structural parameter. No invented entities are introduced.

free parameters (1)
  • Manual topic and theme structure = 20 topics in 8 themes
    Chosen by two researchers when merging BERTopic's 42 generated topics; no formal criterion or validation is reported, and all prevalence percentages depend on this grouping.
assumptions (3)
  • domain assumption YouTube practitioner experience-report videos are a valid proxy for practitioner priorities in building LLM applications.
    The corpus is limited to English-language videos found via five YouTube queries and playlists; the paper cites [8] to argue such videos contain insights comparable to interviews, but no independent validation against a broader practitioner sample is provided.
  • domain assumption Manual relevance filtering of the video pool yields a representative set of experience reports.
    Section III-A states that videos were excluded if not in English or not thematically relevant, with 'thematically relevant' defined as practitioner experience reports; this filter is applied by two researchers from titles, descriptions, and sometimes videos, with no inter-rater reliability or audit reported.
  • ad hoc to paper Paragraphs that BERTopic leaves unclassified are transitions or greetings and carry no topic information.
    Section III-B reports that 1843 of 4100 paragraphs remain unclassified and treats this as expected because videos contain introductions and transitions; this assumption affects every topic prevalence percentage in the results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Themes of Building LLM-based Applications for Production: A Practitioner's View." pith.science (2026). https://pith.science/paper/ZBQCHPMF

@misc{pith2026241108574,
  author       = {Pith},
  title        = {Pith review of: Themes of Building LLM-based Applications for Production: A Practitioner's View},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZBQCHPMF}},
  note         = {Machine review of arXiv:2411.08574}
}
read the original abstract

Background: Large language models (LLMs) have become a paramount interest of researchers and practitioners alike, yet a comprehensive overview of key considerations for those developing LLM-based systems is lacking. This study addresses this gap by collecting and mapping the topics practitioners discuss online, offering practical insights into where priorities lie in developing LLM-based applications. Method: We collected 189 videos from 2022 to 2024 from practitioners actively developing such systems and discussing various aspects they encounter during development and deployment of LLMs in production. We analyzed the transcripts using BERTopic, then manually sorted and merged the generated topics into themes, leading to a total of 20 topics in 8 themes. Results: The most prevalent topics fall within the theme Design & Architecture, with a strong focus on retrieval-augmented generation (RAG) systems. Other frequently discussed topics include model capabilities and enhancement techniques (e.g., fine-tuning, prompt engineering), infrastructure and tooling, and risks and ethical challenges. Implications: Our results highlight current discussions and challenges in deploying LLMs in production. This way, we provide a systematic overview of key aspects practitioners should be aware of when developing LLM-based applications. We further pale off topics of interest for academics where further research is needed.

Figures

Figures reproduced from arXiv: 2411.08574 by the authors.

Figure 1
Figure 1. Overview of the different steps of our applied methodology. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Themes and topics practitioners discuss with regard to LLMs in Production. The size of a box encodes its topic frequency over all videos. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Co-occurrences of topics where darker colors highlight topics occur more often together within individual videos. [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 22 canonical work pages

  1. [1]

    Competition- level code generation with alphacode,

    Y . Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago et al. , “Competition- level code generation with alphacode,” Science, vol. 378, no. 6624, pp. 1092–1097, 2022

    2022

  2. [2]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374 , 2021

    arXiv 2021

  3. [3]

    Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models,

    Y . Deng, C. S. Xia, H. Peng, C. Yang, and L. Zhang, “Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models,” in Proceedings of the 32nd ACM SIGSOFT interna- tional symposium on software testing and analysis , 2023, pp. 423–435

    2023

  4. [4]

    Nuances are the key: Unlocking chatgpt to find failure- inducing tests with differential prompting,

    T.-O. Li, W. Zong, Y . Wang, H. Tian, Y . Wang, S.-C. Cheung, and J. Kramer, “Nuances are the key: Unlocking chatgpt to find failure- inducing tests with differential prompting,” in 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 2023, pp. 14–26

    work page 2023

  5. [5]

    Large language models are few-shot summarizers: Multi-intent comment generation via in-context learning,

    M. Geng, S. Wang, D. Dong, H. Wang, G. Li, Z. Jin, X. Mao, and X. Liao, “Large language models are few-shot summarizers: Multi-intent comment generation via in-context learning,” in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, 2024, pp. 1–13

    2024

  6. [6]

    Navigating challenges and technical debt in large language models deployment,

    A. Menshawy, Z. Nawaz, and M. Fahmy, “Navigating challenges and technical debt in large language models deployment,” in Proceedings of the 4th Workshop on Machine Learning and Systems, 2024, pp. 192–199

    work page 2024

  7. [7]

    Building your own product copilot: Challenges, opportunities, and needs,

    C. Parnin, G. Soares, R. Pandita, S. Gulwani, J. Rich, and A. Z. Henley, “Building your own product copilot: Challenges, opportunities, and needs,” arXiv preprint arXiv:2312.14231 , 2023

    arXiv 2023

  8. [8]

    Socio-technical anti-patterns in building ml-enabled software: Insights from leaders on the forefront,

    A. Mailach and N. Siegmund, “Socio-technical anti-patterns in building ml-enabled software: Insights from leaders on the forefront,” in Proc. Int. Conf. on Software Engineering (ICSE) , 2023, pp. 690–702

    work page 2023

Show all 41 references
  1. [9]

    Collaboration challenges in building ml-enabled systems: Communication, documentation, en- gineering, and process,

    N. Nahar, S. Zhou, G. Lewis, and C. K ¨astner, “Collaboration challenges in building ml-enabled systems: Communication, documentation, en- gineering, and process,” in Proc. Int. Conf. on Software Engineering (ICSE). ACM, 2022, pp. 413–425

    2022

  2. [10]

    Software engineering for machine learning: A case study,

    S. Amershi, A. Begel, C. Bird, R. DeLine, H. Gall, E. Kamar, N. Nagap- pan, B. Nushi, and T. Zimmermann, “Software engineering for machine learning: A case study,” in Proc. Int. Conf. on Software Engineering: Software Engineering in Practice (ICSE-SEIP) . IEEE, 2019, pp. 291– 300

    2019

  3. [11]

    Data scientists in software teams: State of the art and challenges,

    M. Kim, T. Zimmermann, R. DeLine, and A. Begel, “Data scientists in software teams: State of the art and challenges,” Transactions on Software Engineering, vol. 44, no. 11, pp. 1024–1038, 2018

    2018

  4. [12]

    Hidden technical debt in machine learning systems,

    D. Sculley, G. Holt, D. Golovin, E. Davydov, T. Phillips, D. Ebner, V . Chaudhary, M. Young, J.-F. Crespo, and D. Dennison, “Hidden technical debt in machine learning systems,” in Advances in Neural Information Processing Systems, vol. 28. Curran Associates, Inc., 2015, pp. 2503–2511,

    2015

  5. [13]

    How does machine learning change software development practices?

    Z. Wan, X. Xia, D. Lo, and G. C. Murphy, “How does machine learning change software development practices?” IEEE Transactions on Software Engineering , vol. 47, no. 9, pp. 1857–1871, 2019

    2019

  6. [14]

    Software engineering challenges of deep learning,

    A. Arpteg, B. Brinne, L. Crnkovic-Friis, and J. Bosch, “Software engineering challenges of deep learning,” inProc. Euromicro Conference on Software Engineering and Advanced Applications (SEAA) , 2018, pp. 50–59

    2018

  7. [15]

    How do engineers perceive difficulties in engineering of machine-learning systems?-questionnaire survey,

    F. Ishikawa and N. Yoshioka, “How do engineers perceive difficulties in engineering of machine-learning systems?-questionnaire survey,” in 2019 IEEE/ACM Joint 7th International Workshop on Conducting Empirical Studies in Industry (CESI) and 6th International Workshop on Softwa...

    2019

  8. [16]

    A taxonomy of software engineering challenges for machine learning systems: An empirical investigation,

    L. E. Lwakatare, A. Raj, J. Bosch, H. H. Olsson, and I. Crnkovic, “A taxonomy of software engineering challenges for machine learning systems: An empirical investigation,” in Agile Processes in Software Engineering and Extreme Programming: 20th International Conference, XP 201...

    2019

  9. [17]

    A meta- summary of challenges in building products with ml components– collecting experiences from 4758+ practitioners,

    N. Nahar, H. Zhang, G. Lewis, S. Zhou, and C. K ¨astner, “A meta- summary of challenges in building products with ml components– collecting experiences from 4758+ practitioners,” in 2023 IEEE/ACM 2nd International Conference on AI Engineering–Software Engineering for AI (CAIN)...

    2023

  10. [18]

    Prompts are programs too! understanding how developers build software containing prompts,

    J. T. Liang, M. Lin, N. Rao, and B. A. Myers, “Prompts are programs too! understanding how developers build software containing prompts,” arXiv preprint arXiv:2409.12447 , 2024

    2024 arXiv

  11. [19]

    Re- thinking software engineering in the foundation model era: From task- driven ai copilots to goal-driven ai pair programmers,

    A. E. Hassan, G. A. Oliva, D. Lin, B. Chen, Z. Ming et al. , “Re- thinking software engineering in the foundation model era: From task- driven ai copilots to goal-driven ai pair programmers,” arXiv preprint arXiv:2404.10225, 2024

    2024 arXiv

  12. [20]

    Seven failure points when engineering a retrieval augmented generation system,

    S. Barnett, S. Kurniawan, S. Thudumu, Z. Brannelly, and M. Abdelrazek, “Seven failure points when engineering a retrieval augmented generation system,” in Proceedings of the IEEE/ACM 3rd International Conference on AI Engineering-Software Engineering for AI , 2024, pp. 194–199

    2024

  13. [21]

    Llm-based edge intelligence: A comprehensive survey on architectures, applications, security and trustworthiness,

    O. Friha, M. A. Ferrag, B. Kantarci, B. Cakmak, A. Ozgun, and N. Ghoualmi-Zine, “Llm-based edge intelligence: A comprehensive survey on architectures, applications, security and trustworthiness,”IEEE Open Journal of the Communications Society , 2024

    2024

  14. [22]

    Robust speech recognition via large-scale weak supervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” 2022. [Online]. Available: https://arxiv.org/abs/2212.04356

    2022 arXiv

  15. [23]

    Pyannote.audio: Neural building blocks for speaker diarization

    H. Bredin, R. Yin, J. M. Coria, G. Gelly, P. Korshunov, M. Lavechin, D. Fustes, H. Titeux, W. Bouaziz, and M.-P. Gill, “Pyannote.audio: Neural building blocks for speaker diarization.” IEEE, pp. 7124–7128

  16. [24]

    text-splitter,

    B. Brandt, “text-splitter,” https://github.com/benbrandt/text-splitter, 2024, gitHub repository

    2024

  17. [25]

    Bertopic: Neural topic modeling with a class-based tf-idf procedure,

    M. Grootendorst, “Bertopic: Neural topic modeling with a class-based tf-idf procedure,” arXiv preprint arXiv:2203.05794 , 2022

    2022 arXiv

  18. [26]

    Retrieval-augmented generation for large language models: A survey,

    Y . Gao, Y . Xiong, X. Gao, K. Jia, J. Pan, Y . Bi, Y . Dai, J. Sun, M. Wang, and H. Wang, “Retrieval-augmented generation for large language models: A survey,” arXiv preprint arXiv:2312.10997 , 2023

    2023 arXiv

  19. [27]

    A methodology for evaluating rag systems: A case study on configuration dependency validation,

    S. Simon, A. Mailach, J. Dorn, and N. Siegmund, “A methodology for evaluating rag systems: A case study on configuration dependency validation,” arXiv preprint arXiv:2410.08801 , 2024

    2024 arXiv

  20. [28]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685 , 2021

    2021 arXiv

  21. [29]

    Qlora: Efficient finetuning of quantized llms,

    T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: Efficient finetuning of quantized llms,”arXiv preprint arXiv:2305.14314, 2023

    2023 arXiv

  22. [30]

    Exploring hyperparameter usage and tuning in machine learning research,

    S. Simon, N. Kolyada, C. Akiki, M. Potthast, B. Stein, and N. Sieg- mund, “Exploring hyperparameter usage and tuning in machine learning research,” in 2023 IEEE/ACM 2nd International Conference on AI Engineering–Software Engineering for AI (CAIN) . IEEE, 2023, pp. 68–79

    2023

  23. [31]

    Prompt programming for large language models: Beyond the few-shot paradigm,

    L. Reynolds and K. McDonell, “Prompt programming for large language models: Beyond the few-shot paradigm,” in Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems , 2021, pp. 1–7

    2021

  24. [32]

    Scaling laws for neural language models,

    J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,” ArXiv, vol. abs/2001.08361, 2020

    2001 arXiv

  25. [33]

    Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale,

    T. Dettmers, M. Lewis, Y . Belkada, and L. Zettlemoyer, “Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale,”Advances in Neural Information Processing Systems , vol. 35, pp. 30 318–30 332, 2022

    2022

  26. [34]

    A survey on knowledge distillation of large language models,

    X. Xu, M. Li, C. Tao, T. Shen, R. Cheng, J. Li, C. Xu, D. Tao, and T. Zhou, “A survey on knowledge distillation of large language models,” arXiv preprint arXiv:2402.13116 , 2024

    2024 arXiv

  27. [35]

    Who validates the validators? aligning llm-assisted evaluation of llm outputs with human preferences,

    S. Shankar, J. Zamfirescu-Pereira, B. Hartmann, A. Parameswaran, and I. Arawjo, “Who validates the validators? aligning llm-assisted evaluation of llm outputs with human preferences,” in Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, 20...

    2024

  28. [36]

    Openp5: An open-source platform for developing, training, and evaluating llm-based recommender systems,

    S. Xu, W. Hua, and Y . Zhang, “Openp5: An open-source platform for developing, training, and evaluating llm-based recommender systems,” in Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval , 2024, pp. 386– 394

    2024

  29. [37]

    Software engineering using autonomous agents: Are we there yet?

    S. Suri, S. N. Das, K. Singi, K. Dey, V . S. Sharma, and V . Kaulgud, “Software engineering using autonomous agents: Are we there yet?” in 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2023, pp. 1855–1857. VIDEOS V3 MLOps.community ...

    2023

  30. [40]

    Building LLM Applications for Production - AI Campus Berlin

    https://youtube.com/watch?v=BN-txmqGxvQ. V97 AI Campus Berlin “Building LLM Applications for Production - AI Campus Berlin”, YouTube, 2023. https://youtube.com/watch?v= HyucjMv9 -I. V103 MLOps.community “Challenges and Opportunities in Building Data Science Solutions with LLMs...

    2023

  31. [41]

    Building Real-World LLM Products with Fine-Tuning and More with Hamel Husain - 694

    https://youtube.com/watch?v=0j0EtPDunyY. V105 The TWIML AI Podcast with Sam Charrington “Building Real-World LLM Products with Fine-Tuning and More with Hamel Husain - 694”, YouTube, 2024. https://youtube.com/watch?v=sGqEKzJYrNE. V110 FunctionalTV “LLM Avalanche Panel Entepris...

    2024

  32. [2023]

    LLM on K8s // Panel 2 // LLMs in Conference in Production Conference Part 2

    https://youtube.com/watch?v=CJKth2WROVY. V11 MLOps.community “LLM on K8s // Panel 2 // LLMs in Conference in Production Conference Part 2”, YouTube, 2023. https://youtube.com/ watch?v=0e5q4zCBtBs. V15 Snorkel AI “How to tune RAG implementations for special- ized enterprise tas...

    2023

  33. [2024]

    LLMs For the Rest of Us // Vikram Sreekanti & Joseph Gonzalez // LLMs in Prod Conference Part 2

    https://youtube.com/watch?v=ncPCk6wF7l0. V64 MLOps.community “LLMs For the Rest of Us // Vikram Sreekanti & Joseph Gonzalez // LLMs in Prod Conference Part 2”, YouTube, 2023. https://youtube.com/watch?v=AaM6nF1a784. V66 LlamaIndex “LlamaIndex Sessions Practical Tips and Tricks...

    2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.