Pith. sign in

REVIEW 3 major objections 5 minor 114 references

The Hitchhikers Guide to Production-ready Trustworthy Foundation Model powered Software (FMware)

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read FMware's scattered engineering pain points organize into one lifecycle, and a stage-by-stage action plan can move these systems from demos to trustworthy production.

desk verdict A well-organized tutorial survey that repackages the authors' own prior FMware work; useful for practitioners, but the 'systematic' catalogue misses adversarial security and the new-content claim needs clarifying. read the letter →

arxiv 2505.10640 v2 pith:6TTV6VKY submitted 2025-05-15 cs.SE cs.AIcs.LG

Kirill Vasilevski , Benjamin Rombaut , Gopi Krishnan Rajbahadur , Gustavo A. Oliva , Keheliya Gallaba , Filipe R. Cogo , Jiahuei Lin , Dayi Lin
show 5 more authors
Haoxiang Zhang Bouyan Chen Kishanthan Thangarajah Ahmed E. Hassan Zhen Ming Jiang
This is my paper · ORCID
classification cs.SEcs.AIcs.LG
keywords FMwarefoundationmodelslargelanguageAI4SEsoftwareengineeringproductionreadinessLLMlifecycletrustworthyAI
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FMware—software that integrates foundation models as core components—behaves unlike traditional software: outputs are stochastic, dependencies shift with model updates, and compute is expensive, so standard engineering practices break down. The paper argues that the resulting difficulties are not a random list but fall into a lifecycle spanning model selection, data and model alignment, prompting, grounding, agents and orchestration, testing and optimization, deployment and maintenance, with guarding, memory, and compliance as cross-cutting concerns. It pairs each stage with an action plan of concrete directives, some ready to apply and others open research problems, drawn from the authors' industrial experience and recent work. If the roadmap is right, teams gain a shared map of the field and a practical route from impressive demos to production-ready, trustworthy FMware.

What carries the argument

The central object is the FMware lifecycle (Figure 1): a structured view that runs from FM selection through data and model alignment, prompting, grounding, agents and agent orchestration, system testing and optimization, and deployment and maintenance, with guarding, memory management, and compliance as cross-cutting layers. The lifecycle does the argument's work by converting a scattered catalogue of pain points into a shared vocabulary and a stage-by-stage map. The action plan mirrors the lifecycle stage-for-stage, so each directive names the stage it repairs; this one-to-one mapping between challenge and remedy is what gives the roadmap its practical force.

What would settle it

Survey developers of FMware outside large enterprises—startups, open-source maintainers, public-sector teams—and check whether the engineering problems they report map onto the lifecycle stages and directives. A substantial cluster of reported challenges that fits no stage or has no directive would falsify the roadmap's coverage claim.

Watch

Extended reading notes

Core claim

On the paper's own terms, the contribution is the coherent organization itself. It claims that production-ready FMware has a distinct engineering identity: because model behavior changes with prompt wording, inference parameters, model updates, and input distributions, deterministic testing, observability, and compliance assumptions no longer hold. The survey assembles evidence for each lifecycle stage—ad hoc model selection among more than 1.4 million open-source models, brittle and non-portable prompts, hallucination-prone agent orchestration, AI judges correlating only moderately with human judgment (a reported 0.51), static guardrails that paraphrasing bypasses, and compliance tooling that does not span integrated FM components. It then claims these challenges are addressable through the action plan's directives, from semantic caching, prompt versioning, and structured retries to longer-range programs such as multi-level observability, FMwareBOM generation, and formal verification of compliance. A sympathetic reader takes away a structured map and a technology roadmap, not a proof that any single directive works.

Load-bearing premise

The roadmap depends on the curated catalogue of challenges being complete and representative enough to ground its directives; the paper does not show that a list built mainly from large-enterprise industrial experience covers startups, open-source projects, or low-resource deployments.

Editorial extensions

If this is right

  • Teams can use the lifecycle as a pre-production checklist, making sure selection, alignment, prompting, grounding, orchestration, testing, observability, guarding, and compliance are each dealt with before release.
  • Testing must shift from asserting exact outputs to evaluating behavior across distributions, using continuous evaluation pipelines, dependency-aware regression testing, and metamorphic relations derived from user feedback.
  • Compliance needs FMware-wide artifacts: an FMwareBOM that extends SPDX 3.0 AI and dataset profiles to cover code, data, models, and agents, with automated license and regulatory checks.
  • Observability has to become semantic—tracing reasoning paths, agent coordination, and drift—not just collecting latency and token metrics.
  • Agent architecture should follow software-engineering discipline: single-responsibility components, permission models, structured human oversight, and controlled execution snapshots for reproducibility.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the lifecycle is a genuine map, it should pass a classification test: independent practitioners tagging the same set of FMware incidents should agree on which stage each incident belongs to; the paper does not run that validation.
  • The directives mix ready-to-apply tactics with research-level programs, so a natural extension—left implicit—is a maturity ranking that lets teams sequence adoption from low-effort wins such as semantic caching and prompt version control toward harder items such as formal verification and multi-level observability.
  • The reported 0.51 correlation for AI judges suggests an operational rule the paper does not state: treat LLM-as-judge as a triage filter that flags outputs for human review, not as a final oracle.
  • The stage-by-stage structure invites a benchmark suite: each open problem could become a measurable task, such as prompt portability across model families or cache correctness after model updates.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper, written for a KDD 2025 tutorial, presents a survey and roadmap for building trustworthy, production-ready software systems that integrate foundation models as core components, which it terms FMware. It organizes the field into a lifecycle spanning FM selection, alignment, prompting, grounding, agents and orchestration, testing and optimization, observability, guarding, deployment, and cross-cutting concerns such as memory management and compliance. Drawing on industrial anecdotes and prior research, it argues that FMware engineering differs fundamentally from traditional software engineering, then proposes an action plan of directives organized by lifecycle stage. The central claims are that the paper systematically identifies the challenges of FMware and provides a technology roadmap for addressing them.

Significance. If the catalogue of challenges and the roadmap were comprehensive and well-validated, the paper would be a useful shared reference for both researchers and practitioners, synthesizing a fragmented literature and connecting it to concrete production concerns such as LinkedIn's extra four months of deployment effort, CodeCompose's model-sizing trade-offs, and ChatGPT's operating costs. The lifecycle framing and the emphasis on semantic observability, memory management, and compliance are timely and reflect real industrial experience. The paper is also honest about the immaturity of current practice, citing concrete evidence such as the moderate correlation of AI-as-judge evaluation with human judgment. However, the paper's contribution is qualitative and curated rather than systematically derived: none of the twenty-plus directives is empirically evaluated, and the catalogue is not benchmarked against an independently constructed taxonomy, so the roadmap's adequacy depends on the completeness of the authors' own challenge list.

major comments (3)
  1. [§3.5.1 Guarding; §2.7 Cross-cutting concerns; §4 Action Plan] The catalogue omits the entire class of adversarial-input security risks, most notably prompt injection, jailbreaks, indirect injection through retrieved or tool-returned content, data poisoning, and model supply-chain risks. The guarding discussion (§3.5.1) addresses output filtering, semantic filtering, false positives/negatives, and confidence estimation, but never considers that the inputs to an FMware system can be adversarial. Security is also absent from the cross-cutting concerns (§2.7), which list only Memory Management and Compliance, and the action plan (§4) contains no security directive. Because the paper's central claim in §1 is that it 'systematically identifies these challenges' and provides a production-readiness roadmap, this omission is load-bearing: teams following the roadmap would receive no guidance on what is widely regarded as one of the most critical production risk classes for LLM-based systems, corresponding to the top OWASP LLM risk. This should be remedied by adding a dedicated security subsection or cross-cutting concern, with corresponding directives, and by discussing the relationship between guarding and adversarial input robustness.
  2. [§1 Introduction; §2 FMware Lifecycle; §3 Challenges] The paper claims to 'systematically identify' the challenges of FMware, but it provides no survey methodology: no search strategy, inclusion/exclusion criteria, source selection process, or procedure for validating the representativeness of the challenge set. The catalogue appears to be heavily drawn from the authors' prior works [43], [76], [44], [57], and [30], and the paper does not compare its taxonomy with an independently derived one. This makes it impossible to assess whether the challenge list is comprehensive beyond the authors' large-enterprise industrial context, or whether the priority ordering of the roadmap would transfer to startups, open-source projects, low-resource deployments, or non-enterprise domains. The completeness of the catalogue is load-bearing for the roadmap claim, so the paper should either add a methodology subsection describing how the catalogue was assembled and validated, or soften the 'systematically identifies' claim to reflect the curated and experience-based nature of the survey.
  3. [§4 Action Plan] The action plan presents more than twenty directives (e.g., 'Multi-Prompt Architectures and Human-in-the-Loop Workflows', 'Automated FMwareBOM Generation and Formal Verification', 'Resource-Aware QA Framework and Smart Caching') without any evidence that these directives are effective, feasible, or prioritized relative to one another. The paper states that 'some directives can be somewhat readily applied to mitigate challenges, whereas others are research topics in themselves', but it does not say which are which, nor does it provide any case study, pilot, or empirical assessment. As a tutorial roadmap the proposals may be reasonable, but the manuscript's contribution framing in §1 — a roadmap for addressing the challenges — would be strengthened by explicitly marking each directive as validated practice, emerging practice, or open research, and by discussing any known limitations or trade-offs.
minor comments (5)
  1. [§1 Introduction] There is a grammatical slip in 'We then discuss key the challenges and present a critical overview' — 'key the' should be 'the key'.
  2. [§2.1, §2.2, §2.3] Several sentences use informal constructions, such as 'Below we discuss about the different stages' and 'we henceforth present an action plan'; the paper would benefit from a light language pass for consistency with a formal proceedings style.
  3. [Author block] The author list contains a formatting error: 'Gustavo A. Oliva1,2,,' has a doubled comma, and affiliation markers are inconsistently applied across the author list.
  4. [References] A few references are missing author or venue details, for example [79] lists 'RobBagby' as the author and [88] lists 'Unknown'; these should be completed for a camera-ready version.
  5. [Figure 1] The lifecycle figure uses the labels 'Agent(s)' and 'Agent orchestration' as separate boxes while the text in §2.3 discusses both under a single heading; the visual relationship between these boxes and the 'Guarding' and 'Memory management' boxes would be clearer if the figure legend explained the cross-cutting vs. sequential nature of the stages.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation found: the paper is a survey/tutorial that organizes existing evidence and cites the authors' prior work as background, not as the sole justification for its claims.

full rationale

This paper is a tutorial and survey, not a derivation with fitted parameters or predictive equations. Its central claims are that FMware poses distinctive engineering challenges and that those challenges can be organized into a lifecycle plus an action plan. Those claims are supported by external industrial and academic evidence (e.g., LinkedIn's deployment delays [2], Microsoft/GitHub validation costs [72], ChatGPT operating costs [7], Hugging Face model counts [32], and independently cited work on testing, alignment, RAG, agents, and compliance). The authors' own prior publications [43, 76] are cited where the paper explicitly builds on or condenses that previous catalogue and roadmap, which is normal attribution for a survey. Other same-group citations ([30], [44], [57]) appear as pointers to proposed tools in the roadmap; they do not function as a self-referential proof that the roadmap is correct. No equation or claim reduces by construction to its own input: the lifecycle is a proposed organizational framework, the challenges are curated from the literature and industrial reports, and the action plan is explicitly 'driven by the surveyed challenges' plus practical experience. Concerns about completeness (e.g., no dedicated prompt-injection or adversarial-security stage) and about the absence of a formal survey methodology are correctness or scope risks for the roadmap, but they are not circularity under the stated criteria. Accordingly, no circular step can be quoted, and the paper warrants a score of 0 for circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 3 invented entities

The paper makes no numeric fits, so there are no free parameters. The conceptual content rests on the axioms above: that FMware is a distinct class, that the challenge catalogue is comprehensive, that cited industrial anecdotes are accurate, and that conventional SE tools transfer. It also introduces several new labeled concepts and artifacts (FMware, FMwareBOM, semantic observability) without independent falsifiable handles. These ledger entries quantify how much of the paper is repackaging versus new, tested contribution.

assumptions (4)
  • domain assumption FMware is a categorically distinct class of software requiring engineering practices that differ fundamentally from traditional software development.
    The entire paper's need for a new lifecycle and roadmap rests on this categorical distinction, argued in Section 1 by examples and citations ([43], [76]) rather than by a comparative empirical study of traditional vs FMware projects.
  • domain assumption The curated challenge catalogue is comprehensive and representative across FMware contexts.
    The authors state the survey 'provides a cohesive overview of all challenges related to FMware development' (Section 1) but provide no selection methodology or external validation for this completeness.
  • domain assumption Cited industry anecdotes and statistics are accurate and generalizable (LinkedIn deployment delay, Microsoft/GitHub validation costs, ChatGPT $700k/day, AI-as-judge correlation 0.51).
    These anecdotes motivate the problem statement (Section 1) and the testing discussion (Section 3.4.2) but are taken at face value from blog posts, industry reports, and a single evaluation study.
  • domain assumption Established software engineering principles (modularity, single responsibility, version control, formal verification) can be transferred to FMware.
    The action plan (Section 4) directs practitioners to adopt established software engineering principles and use SMT-based formal verification for compliance; whether these tools work on open-ended, stochastic FM behavior is asserted, not demonstrated.
invented entities (3)
  • FMware
    purpose: Organizing construct for the entire paper; names the class of systems that integrate FMs as core components.
    Defined in Section 1 and reused throughout. The term originates in the authors' prior work [43]; no falsifiable observable is attached to the term itself beyond the rhetorical framing.
  • FMwareBOM
    purpose: Proposed automation artifact for compliance: a bill of materials for FMware components, extending SPDX 3.0 with models, datasets, synthetic data, RLHF data, and user feedback.
    Proposed in Section 4.5 ('Automated FMwareBOM Generation and Formal Verification'). No implementation, schema, or evaluation is provided in this paper.
  • Semantic observability
    purpose: Extends traditional observability to capture user corrections, intent recognition, and model drift in FMware.
    Introduced in Section 2.4 as FMware-specific observability; the paper describes the concept but provides no tool or measurement. The term is carried over from the authors' prior catalogue [43].

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Hitchhikers Guide to Production-ready Trustworthy Foundation Model powered Software (FMware)." pith.science (2026). https://pith.science/paper/6TTV6VKY

@misc{pith2026250510640,
  author       = {Pith},
  title        = {Pith review of: The Hitchhikers Guide to Production-ready Trustworthy Foundation Model powered Software (FMware)},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6TTV6VKY}},
  note         = {Machine review of arXiv:2505.10640}
}
read the original abstract

Foundation Models (FMs) such as Large Language Models (LLMs) are reshaping the software industry by enabling FMware, systems that integrate these FMs as core components. In this KDD 2025 tutorial, we present a comprehensive exploration of FMware that combines a curated catalogue of challenges with real-world production concerns. We first discuss the state of research and practice in building FMware. We further examine the difficulties in selecting suitable models, aligning high-quality domain-specific data, engineering robust prompts, and orchestrating autonomous agents. We then address the complex journey from impressive demos to production-ready systems by outlining issues in system testing, optimization, deployment, and integration with legacy software. Drawing on our industrial experience and recent research in the area, we provide actionable insights and a technology roadmap for overcoming these challenges. Attendees will gain practical strategies to enable the creation of trustworthy FMware in the evolving technology landscape.

Figures

Figures reproduced from arXiv: 2505.10640 by the authors.

Figure 1
Figure 1. FMware Lifecycle. 2.5 Performance Engineering, Testing, and Optimization FMware validation differs from traditional software testing due to its non-deterministic outputs. Unlike conventional applications, where the same input produces the same output, FMware responses vary based on model updates, prompt changes, and shifting input distributions [63]. Ensuring reliability requires continuous evalua￾tion pipelines tha… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

114 extracted references · 20 canonical work pages

  1. [43]

    Ahmed E. Hassan, Dayi Lin, Gopi Krishnan Rajbahadur, Keheliya Gallaba, Fil- ipe Roseiro Cogo, Boyuan Chen, Haoxiang Zhang, Kishanthan Thangarajah, Gustavo Oliva, Jiahuei (Justina) Lin, Wali Mohammad Abdullah, and Zhen Ming (Jack) Jiang. 2024. Rethinking Software Engineering in the Era of Foun- dation Models: A Curated Catalogue of Challenges in the Develo...

    2024

  2. [76]

    Oliva, Dayi Lin, and Ahmed E

    Gopi Krishnan Rajbahadur, Gustavo A. Oliva, Dayi Lin, and Ahmed E. Hassan

    work page

  3. [44]

    Hassan, Gustavo A

    Ahmed E. Hassan, Gustavo A. Oliva, Dayi Lin, Boyuan Chen, Zhen Ming, and Jiang. 2024. Towards AI-Native Software Engineering (SE 3.0): A Vision and a Challenge Roadmap. arXiv:2410.06107

    arXiv 2024

  4. [57]

    Jiahuei Lin, Dayi Lin, Sky Zhang, and Ahmed E. Hassan. 2024. Engineering AI Judge Systems. arXiv:2411.17793 [cs.SE] https://arxiv.org/abs/2411.17793

    arXiv 2024

  5. [30]

    Ximing Dong, Dayi Lin, Shaowei Wang, and Ahmed E Hassan. 2024. A Frame- work for Real-time Safeguarding the Text Generation of Large Language.arXiv preprint arXiv:2404.19048(2024)

    arXiv 2024

  6. [1]

    Testing LLM-Based Applications: Strategy and Challenges — blog.scottlogic.com

    2023. Testing LLM-Based Applications: Strategy and Challenges — blog.scottlogic.com. https://blog.scottlogic.com/2023/11/14/testing-LLM-based- applications-strategy-and-challenges.html. [Accessed 07-10-2024]

    2023

  7. [2]

    Musings on Building a Generative AI Product — linkedin.com

    2024. Musings on Building a Generative AI Product — linkedin.com. https://www.linkedin.com/blog/engineering/generative-ai/musings-on- building-a-generative-ai-product. [Accessed 08-10-2024]

    2024

  8. [3]

    Copilot Internals

    2025. Copilot Internals. https://thakkarparth007.github.io/copilot-explorer/ posts/copilot-internals last accessed: 2025-03-11

    2025

Show all 114 references
  1. [4]

    LangSmith by LangChain

    2025. LangSmith by LangChain. https://www.langchain.com/langsmith/ last accessed: 2025-03-11

    2025

  2. [5]

    Traceloop

    2025. Traceloop. https://www.traceloop.com/ last accessed: 2025-03-11

    2025

  3. [6]

    Weights and Biases - Weave

    2025. Weights and Biases - Weave. https://wandb.ai/site/weave/ last accessed: 2025-03-11

    2025

  4. [7]

    Assad Abbas

    Dr. Assad Abbas. 2024. The Financial Challenges of Leading in AI: A Look at OpenAI’s Operating Costs — unite.ai. https://www.unite.ai/the-financial- challenges-of-leading-in-ai-a-look-at-openais-operating-costs/. [Accessed 11-10-2024]

    2024

  5. [8]

    Toufique Ahmed and Premkumar Devanbu. 2023. Better patching using LLM prompting, via Self-Consistency. In2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 1742–1746

    2023

  6. [9]

    Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, et al. 2024. A survey on data selection for language models.arXiv preprint arXiv:2402.16827(2024)

    2024 arXiv

  7. [10]

    Oluyemi Enoch Amujo and Shanchieh Jay Yang. 2024. How Good Is It? Evaluat- ing the Efficacy of Common versus Domain-Specific Prompts on Foundational Large Language Models.arXiv preprint arXiv:2407.11006(2024)

    2024 arXiv

  8. [11]

    Anthropic. n.d.. All models overview - Anthropic. https://docs.anthropic.com/ en/docs/about-claude/models/all-models. Accessed: 2025-02-01

    2025

  9. [12]

    Anthropic. n.d.. Claude 3.7 Sonnet and Claude Code. https://www.anthropic. com/news/claude-3-7-sonnet. Accessed: 2025-02-01

    2025

  10. [13]

    Narges Ashtari, Ryan Mullins, Crystal Qian, James Wexler, Ian Tenney, and Mahima Pushkarna. 2023. From Discovery to Adoption: Understanding the ML Practitioners’ Interpretability Journey. InProceedings of the 2023 ACM Designing Interactive Systems Conference. 2304–2325

    2023

  11. [14]

    Abhaya Asthana and Jack Olivieri. 2009. Quantifying software reliability and readiness. In2009 IEEE international workshop technical committee on communi- cations quality and reliability. IEEE, 1–6

    2009

  12. [15]

    Suriya Ganesh Ayyamperumal and Limin Ge. 2024. Current state of LLM Risks and AI Guardrails.arXiv preprint arXiv:2406.12934(2024)

    2024 arXiv

  13. [16]

    Bommasani, D.A Hudson, E

    R. Bommasani, D.A Hudson, E. Adeli, Altman, et al. 2021. On the opportunities and risks of foundation models.arXiv preprint arXiv:2108.07258(2021)

    2021 arXiv

  14. [17]

    Brajovic, N

    D. Brajovic, N. Renner, V.P. Goebels, P. Wagner, B. Fresz, et al . 2023. Model Reporting for Certifiable AI: A Proposal from Merging EU Regulation into AI Development.arXiv preprint arXiv:2307.11525(2023)

    2023 arXiv

  15. [18]

    Eric Breck, Shanqing Cai, Eric Nielsen, Michael Salib, and D Sculley. 2017. The ML test score: A rubric for ML production readiness and technical debt reduction. In2017 IEEE international conference on big data (big data). IEEE, 1123–1132

    2017

  16. [19]

    Cramer, S

    K.K Chang, M. Cramer, S. Soni, and D. Bamman. 2023. Speak, memory: An archaeology of books known to chatgpt/gpt-4.arXiv preprint arXiv:2305.00118 (2023)

    2023 arXiv

  17. [20]

    Dong Chen, Shaoxin Lin, Muhan Zeng, Daoguang Zan, Jian-Gang Wang, An- ton Cheshkov, Jun Sun, Hao Yu, Guoliang Dong, Artem Aliev, et al . 2024. CodeR: Issue Resolving with Multi-Agent and Task Graphs.arXiv preprint arXiv:2406.01304(2024)

    2024 arXiv

  18. [21]

    Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang

  19. [22]

    Hung-Ting Chen, Michael JQ Zhang, and Eunsol Choi. 2022. Rich knowledge sources bring complex knowledge conflicts: Recalibrating models to reflect conflicting evidence.arXiv preprint arXiv:2210.13701(2022)

    2022 arXiv

  20. [23]

    L. Chen, M. Zaharia, and J. Zou. 2023. How is ChatGPT’s behavior changing over time? arXiv:2307.09009 [cs.CL]

    2023 arXiv

  21. [24]

    Xiang Chen, Chaoyang Gao, Chunyang Chen, Guangbei Zhang, and Yong Liu

  22. [25]

    Contractor, D

    D. Contractor, D. McDuff, J.K. Haines, J. Lee, C. Hines, et al. 2022. Behavioral use licensing for responsible AI. InProceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency. 778–788

    2022

  23. [26]

    Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. 2024. The power of noise: Redefining retrieval for rag systems. InProceedings of the 47th International ACM SIGIR Conference on...

    2024

  24. [27]

    James Cusick. 2013. Architecture and Production Readiness Reviews in Practice. arXiv preprint arXiv:1305.2402(2013)

    2013 arXiv

  25. [28]

    M. Deng, J. Wang, C.-P. Hsieh, Y. Wang, H. Guo, et al. 2022. RLPrompt: Opti- mizing Discrete Text Prompts with Reinforcement Learning. InProc. of Conf. on EMNLP. ACL, 3369–3391

    2022

  26. [29]

    Hao Ding, Ziwei Fan, Ingo Guehring, Gaurav Gupta, Wooseok Ha, Jun Huan, Linbo Liu, Behrooz Omidvar-Tehrani, Shiqi Wang, and Hao Zhou. 2024. Rea- soning and Planning with Large Language Models in Code Development. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Di...

    2024

  27. [31]

    L. Edwards. 2021. The EU AI Act: a summary of its significance and scope.AI (the EU AI Act)1 (2021)

    2021

  28. [32]

    Hugging Face. n.d.. Models - Hugging Face. https://huggingface.co/models. Accessed: 2025-02-01

    2025

  29. [33]

    Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M Zhang. 2023. Large language models for software engineer- ing: Survey and open problems. In2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineer...

    2023

  30. [34]

    Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. 2023. Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution. arXiv:2309.16797 [cs.CL] https://arxiv.org/abs/2309.16797

    2023 arXiv

  31. [35]

    T. Gao, A. Fisch, and D. Chen. 2021. Making Pre-trained Language Models Better Few-shot Learners. InProc. of Annual Meeting of the ACL-IJCNLP. (ACL), 3816–3830

    2021

  32. [36]

    I. Gim, G. Chen, S.S. Lee, N. Sarda, A. Khandelwal, et al . 2023. Prompt Cache: Modular Attention Reuse for Low-Latency Inference.arXiv preprint arXiv:2311.04934(2023)

    2023 arXiv

  33. [37]

    Github. [n. d.]. GitHub Next | Copilot Workspace. https://githubnext.com/ projects/copilot-workspace/. Accessed 02-06-2024

    2024

  34. [38]

    Grand View Research. 2023. Large Language Model Market Size, Share & Trends Analysis Report By Application (Customer Service, Content Generation), By Deployment, By Industry Vertical, By Region, And Segment Forecasts, 2024 -

    2023

  35. [39]

    Grosse, J

    R. Grosse, J. Bae, C. Anil, N. Elhage, A. Tamkin, et al . 2023. Studying large language model generalization with influence functions.arXiv preprint arXiv:2308.03296(2023)

    2023 arXiv

  36. [40]

    Gunasekar, Y

    S. Gunasekar, Y. Zhang, J. Aneja, C.C.T. Mendes, A. Del Giorno, et al . 2023. Textbooks Are All You Need.arXiv preprint arXiv:2306.11644(2023)

    2023 arXiv

  37. [41]

    Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. 2024. Large language model based multi-agents: A survey of progress and challenges.arXiv preprint arXiv:2402.01680(2024)

    2024 arXiv

  38. [42]

    Tiezheng Guo, Qingwen Yang, Chen Wang, Yanyi Liu, Pan Li, Jiawei Tang, Dapeng Li, and Yingyou Wen. 2024. Knowledgenavigator: Leveraging large language models for enhanced reasoning over knowledge graph.Complex & Intelligent Systems10, 5 (2024), 7063–7076

    2024

  39. [45]

    S. Hong, M. Zhuge, J. Chen, X. Zheng, Y. Cheng, et al. 2023. MetaGPT: Meta Pro- gramming for A Multi-Agent Collaborative Framework. arXiv:2308.00352 [cs.AI]

    2023 arXiv

  40. [46]

    Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large language models for software engineering: A systematic literature review.ACM Transactions on Software Engineering and Methodology33, 8 (2024), 1–79

    2024

  41. [47]

    Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang, Hao Wang, Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen. 2024. Understanding the planning of LLM agents: A survey.arXiv preprint arXiv:2402.02716(2024)

    2024 arXiv

  42. [48]

    Sablayrolles, A

    A.Q Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D.S. Chaplot, et al . 2023. Mistral 7B.arXiv preprint arXiv:2310.06825(2023)

    2023 arXiv

  43. [49]

    Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024. A Survey on Large Language Models for Code Generation.arXiv preprint arXiv:2406.00515(2024)

    2024 arXiv

  44. [50]

    Haolin Jin, Linghan Huang, Haipeng Cai, Jun Yan, Bo Li, and Huaming Chen

  45. [51]

    Uday Kamath, Kevin Keenan, Garrett Somers, and Sarah Sorenson. 2024. LLMs in Production. InLarge Language Models: A Deep Dive: Bridging Theory and Practice. Springer, 315–373

    2024

  46. [52]

    Ryo Kamoi, Sarkar Snigdha Sarathi Das, Renze Lou, Jihyun Janice Ahn, Yilun Zhao, Xiaoxin Lu, Nan Zhang, Yusen Zhang, Ranran Haoran Zhang, Su- jeeth Reddy Vummanthala, et al. 2024. Evaluating LLMs at Detecting Errors in LLM Responses.arXiv preprint arXiv:2404.03602(2024)

    2024 arXiv

  47. [53]

    Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Ke- shav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts

  48. [54]

    KDD ’25, August 3–7, 2025, Toronto, ON, Canada Vasilevski et al

    From llms to llm-based agents for software engineering: A survey of current, challenges and future.arXiv preprint arXiv:2408.02479(2024). KDD ’25, August 3–7, 2025, Toronto, ON, Canada Vasilevski et al

    2024 arXiv

  49. [55]

    Philippe Laban, Alexander R Fabbri, Caiming Xiong, and Chien-Sheng Wu. 2024. Summary of a haystack: A challenge to long-context llms and rag systems.arXiv preprint arXiv:2407.01370(2024)

    2024 arXiv

  50. [56]

    Lewis, E

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, et al . 2020. Retrieval- Augmented Generation for Knowledge-Intensive NLP Tasks. InAdvances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran...

    2020

  51. [58]

    DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines.The Twelfth International Conference on Learning Representations

  52. [59]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. InProceedings of the 29th Symposium on Operating Systems Principl...

    2023

  53. [60]

    Y. Liu, D. Iter, Yi. Xu, S. Wang, R. Xu, and C. Zhu. 2023. Gpteval: Nlg evaluation using gpt-4 with better human alignment.arXiv preprint arXiv:2303.16634 (2023)

    2023 arXiv

  54. [61]

    Longpre, R

    S. Longpre, R. Mahari, A. Chen, N. Obeng-Marnu, D. Sileo, et al. 2023. The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI.arXiv preprint arXiv:2310.16787(2023)

    2023 arXiv

  55. [62]

    Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Qingwei Lin, Jianguang Lou, Shifeng Chen, Yansong Tang, and Weizhu Chen. 2024. Arena Learning: Build Data Flywheel for LLMs Post-training via Simulated Chatbot Arena.arXiv preprint arXiv:2407.10627(2024)

    2024 arXiv

  56. [63]

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts.Transactions of the Association for Computational Linguistics 12 (2024), 157–173

    2024

  57. [64]

    Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, et al. 2024. Best practices and lessons learned on synthetic data for language models.arXiv preprint arXiv:2404.07503(2024)

    2024 arXiv

  58. [65]

    Mozes, X

    M. Mozes, X. He, B. Kleinberg, and L.D Griffin. 2023. Use of llms for illicit purposes: Threats, prevention measures, and vulnerabilities.arXiv preprint arXiv:2308.12833(2023)

    2023 arXiv

  59. [66]

    Murali, C

    V. Murali, C. Maddila, I. Ahmad, M. Bolin, D. Cheng, et al. 2023. CodeCompose: A Large-Scale Industrial Deployment of AI-assisted Code Authoring.arXiv preprint arXiv:2305.12050(2023)

    2023 arXiv

  60. [67]

    Nadia Nahar, Christian Kästner, Jenna Butler, Chris Parnin, Thomas Zimmer- mann, and Christian Bird. 2024. Beyond the Comfort Zone: Emerging Solutions to Overcome Challenges in Integrating LLMs into Software Products.arXiv preprint arXiv:2410.12071(2024)

    2024 arXiv

  61. [68]

    Wanqin Ma, Chenyang Yang, and Christian Kästner. 2024. (Why) Is My Prompt Getting Worse? Rethinking Regression Testing for Evolving LLM APIs. InPro- ceedings of the IEEE/ACM 3rd International Conference on AI Engineering-Software Engineering for AI. 166–171

    2024

  62. [69]

    Microsoft. [n. d.]. microsoft/semantic-kernel: Integrate cutting-edge LLM technology quickly and easily into your apps. https://github.com/microsoft/ semantic-kernel. Accessed 01-30-2024

    2024

  63. [70]

    OpenAI. n.d.. OpenAI o1 Hub | OpenAI. https://openai.com/o1/. Accessed: 2025-02-01

    2025

  64. [71]

    Charles Packer, Vivian Fang, Shishir G Patil, Kevin Lin, Sarah Wooders, and Joseph E Gonzalez. 2023. Memgpt: Towards llms as operating systems.arXiv preprint arXiv:2310.08560(2023)

    2023 arXiv

  65. [72]

    Parnin, G

    C. Parnin, G. Soares, R. Pandita, S. Gulwani, J. Rich, et al . 2023. Build- ing Your Own Product Copilot: Challenges, Opportunities, and Needs. arXiv:2312.14231 [cs.SE]

    2023 arXiv

  66. [73]

    OpenAI. n.d.. Hello GPT-4o | OpenAI. https://openai.com/index/hello-gpt-4o/. Accessed: 2025-02-01

    2025

  67. [74]

    OpenAI. n.d.. Models - OpenAI API. https://platform.openai.com/docs/models. Accessed: 2025-02-01

    2025

  68. [75]

    Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qi- uyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, and Jianfeng Gao. 2023. Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback. arXiv:2302.12813 [c...

    2023 arXiv

  69. [77]

    Sumedh Rasal and EJ Hauer. 2024. Navigating Complexity: Orchestrated Problem Solving with Multi-Agent LLMs.arXiv preprint arXiv:2402.16713(2024)

    2024 arXiv

  70. [78]

    Parry, G.M

    O. Parry, G.M. Kapfhammer, M. Hilton, and P. McMinn. 2022. Surveying the developer experience of flaky tests. InProc. of Int. Conf. on Softw. Eng.: Software Engineering in Practice. 253–262

    2022

  71. [79]

    Venkatesh Balavadhani Parthasarathy, Ahtsham Zafar, Aafaq Khan, and Ar- salan Shahid. 2024. The Ultimate Guide to Fine-Tuning LLMs from Basics to Breakthroughs: An Exhaustive Review of Technologies, Research, Best Practices, Applied Research Challenges and Opportunities.arXiv ...

    2024 arXiv

  72. [80]

    Antonio Sabbatella, Andrea Ponti, Ilaria Giordani, Antonio Candelieri, and Francesco Archetti. 2024. Prompt Optimization in Large Language Models. Mathematics12, 6 (2024), 929

    2024

  73. [81]

    Kunal Sawarkar, Abhilasha Mangal, and Shivam Raj Solanki. 2024. Blended RAG: Improving RAG (Retriever-Augmented Generation) Accuracy with Semantic Search and Hybrid Query-Based Retrievers.arXiv preprint arXiv:2404.07220 (2024)

    2024 arXiv

  74. [82]

    arXiv:2410.20791 [cs.SE] https://arxiv.org/abs/2410.20791

    From Cool Demos to Production-Ready FMware: Core Challenges and a Technology Roadmap. arXiv:2410.20791 [cs.SE] https://arxiv.org/abs/2410.20791

    arXiv

  75. [83]

    Aizaz Sharif, Dusica Marijan, and Marius Liaaen. 2021. Deeporder: Deep learn- ing for test case prioritization in continuous integration testing. In2021 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 525–534

    2021

  76. [84]

    Traian Rebedea, Razvan Dinu, Makesh Sreedhar, Christopher Parisien, and Jonathan Cohen. 2023. Nemo guardrails: A toolkit for controllable and safe llm applications with programmable rails.arXiv preprint arXiv:2310.10501(2023)

    2023 arXiv

  77. [85]

    RobBagby. [n. d.]. Retry Storm antipattern - Performance antipatterns for cloud apps — learn.microsoft.com. https://learn.microsoft.com/en-us/azure/ architecture/antipatterns/retry-storm/. [Accessed 07-10-2024]

    2024

  78. [86]

    Strobelt, A

    H. Strobelt, A. Webson, V. Sanh, B. Hoover, J. Beyer, et al. 2023. Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Models.IEEE Trans. on Visualization and Computer Graphics29, 1 (2023), 1146– 1156

    2023

  79. [87]

    Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, and Dieuwke Hupkes. 2024. Judging the Judges: Evaluating Align- ment and Vulnerabilities in LLMs-as-Judges.arXiv preprint arXiv:2406.12624 (2024)

    2024 arXiv

  80. [88]

    Schoch, R

    S. Schoch, R. Mishra, and Y. Ji. 2023. Data Selection for Fine-tuning Large Lan- guage Models Using Transferred Shapley Values.arXiv preprint arXiv:2306.10165 (2023)

    2023 arXiv

  81. [89]

    Cangqing Wang, Yutian Yang, Ruisi Li, Dan Sun, Ruicong Cai, Yuzhu Zhang, and Chengqian Fu. 2024. Adapting llms for efficient context processing through soft prompt compression. InProceedings of the International Conference on Modeling, Natural Language Processing and Machine L...

    2024

  82. [90]

    Tianhao Shen, Renren Jin, Yufei Huang, Chuang Liu, Weilong Dong, Zishan Guo, Xinwei Wu, Yan Liu, and Deyi Xiong. 2023. Large language model alignment: A survey.arXiv preprint arXiv:2309.15025(2023)

    2023 arXiv

  83. [91]

    W. Shi, A. Ajith, M. Xia, Y. Huang, D. Liu, et al. 2023. Detecting pretraining data from large language models.arXiv preprint arXiv:2310.16789(2023)

    2023 arXiv

  84. [92]

    Xin Wang, Yuwei Zhou, Hong Chen, and Wenwu Zhu. 2024. Curriculum Learn- ing: Theories, Approaches, Applications, Tools, and Future Directions in the Era of Large Language Models. InCompanion Proceedings of the ACM on Web Conference 2024. 1306–1310

    2024

  85. [93]

    Yuntao Wang, Yanghe Pan, Miao Yan, Zhou Su, and Tom H Luan. 2023. A survey on ChatGPT: AI-generated contents, challenges, and solutions.IEEE Open Journal of the Computer Society(2023)

    2023

  86. [94]

    Unknown. 2023. SPDX 3.0 Dataset Profile. https://spdx.github.io/spdx-spec/v3. 0/model/Dataset/Dataset/ Accessed: 2024-10-11

    2023

  87. [95]

    OpenDataology workgroup. 2022. OpenDataology. https://github.com/ OpenDataology/. Accessed 02-07-2024

    2022

  88. [96]

    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2023. Voyager: An open-ended embod- ied agent with large language models.arXiv preprint arXiv:2305.16291(2023)

    2023 arXiv

  89. [97]

    Qiaosi Wang, Koustuv Saha, Eric Gregori, David Joyner, and Ashok Goel. 2021. Towards mutual theory of mind in human-ai interaction: How language reflects what students perceive about a virtual teaching assistant. InProceedings of the 2021 CHI conference on human factors in com...

    2021

  90. [98]

    Yuchen Xia, Jiho Kim, Yuhan Chen, Haojie Ye, Souvik Kundu, Nishil Talati, et al. 2024. Understanding the Performance and Estimating the Cost of LLM The Hitchhikers Guide to Production-ready Trustworthy Foundation Model powered Software (FMware) KDD ’25, August 3–7, 2025, Toron...

    2024 arXiv

  91. [99]

    Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. 2023. Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts.arXiv preprint arXiv:2305.13300(2023)

    2023 arXiv

  92. [100]

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models.Advances in Neural Info Processing Systems(2022)

    2022

  93. [101]

    Quanjun Zhang, Chunrong Fang, Yang Xie, Yaxin Zhang, Yun Yang, Weisong Sun, Shengcheng Yu, and Zhenyu Chen. 2023. A survey on large language models for software engineering.arXiv preprint arXiv:2312.15223(2023)

    2023 arXiv

  94. [102]

    Q. Wu, G. Bansal, J. Zhang, Y. Wu, B. Li, E. Zhu, et al . 2023. Auto- Gen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation. arXiv:2308.08155 [cs.AI]

    2023 arXiv

  95. [103]

    Siye Wu, Jian Xie, Jiangjie Chen, Tinghui Zhu, Kai Zhang, and Yanghua Xiao

  96. [104]

    How Easily do Irrelevant Inputs Skew the Responses of Large Language Models?arXiv preprint arXiv:2404.03302(2024)

    2024 arXiv

  97. [105]

    Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023. Large Language Models Are Human-Level Prompt Engineers. arXiv:2211.01910 [cs.LG] https://arxiv.org/abs/2211.01910

    2023 arXiv

  98. [106]

    X. Zhu, J. Li, Y. Liu, C. Ma, and W. Wang. 2023. A survey on model compression for large language models.arXiv preprint arXiv:2308.07633(2023)

    2023 arXiv

  99. [107]

    Eugene Yan, Bryan Bischof, Charles Frye, Hamel Husain, Jason Liu, and Shreya Shankar. [n. d.]. What We Learned from a Year of Building with LLMs (Part I) — oreilly.com. https://www.oreilly.com/radar/what-we-learned-from-a-year-of- building-with-llms-part-i/. [Accessed 04-10-2024]

    2024

  100. [109]

    Zhimin Zhao, Abdul Ali Bangash, Filipe Roseiro Côgo, Bram Adams, and Ahmed E. Hassan. 2025. On the Workflows and Smells of Leaderboard Opera- tions (LBOps): An Exploratory Study of Foundation Model Leaderboards.IEEE Transactions on Software Engineering(2025), 1–18. doi:10.1109...

    2025

  101. [110]

    C. Zhou, P. Liu, P. Xu, S. Iyer, J. Sun, et al. 2023. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206(2023)

    2023 arXiv

  102. [111]

    Zhou, Y.E

    W. Zhou, Y.E. Jiang, L. Li, J. Wu, T. Wang, et al. 2023. Agents: An Open-source Framework for Autonomous Language Agents. arXiv:2309.07870 [cs.CL]

    2023 arXiv

  103. [114]

    Małgorzata Łazuka, Andreea Anghel, and Thomas Parnell. 2024. LLM-Pilot: Characterize and Optimize Performance of your LLM Inference Services. arXiv:2410.02425 https://arxiv.org/abs/2410.02425

    2024 arXiv

  104. [2024]

    Humans or llms as the judge? a study on judgement biases.arXiv preprint arXiv:2402.10669(2024)

    2024 arXiv

  105. [2025]

    An Empirical Study on Challenges for LLM Application Developers.ACM Transactions on Software Engineering and Methodology(2025)

    2025

  106. [2030]

    https://www.grandviewresearch.com/industry-analysis/large-language- model-llm-market-report

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.