Pith. sign in

REVIEW 1 major objections 1 minor 1 cited by

Querying Databases with Function Calling

T0 review · 1 major / 1 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Function Calling provides an effective and generalizable interface for natural language database access.

desk verdict A credible, transparent benchmark for LLM database querying via function calling, but the synthetic GPT-4o-generated data leaves generalization unproven. read the letter →

arxiv 2502.00032 v1 pith:XHOZYFU2 submitted 2025-01-23 cs.DB cs.AIcs.IR

Connor Shorten , Charles Pierse , Thomas Benjamin Smith , Karel D'Oosterlinck , Tuana Celik , Erika Cardenas , Leonie Monigatti , Mohd Shukri Hasan
show 4 more authors
Edward Schmuhl Daniel Williams Aravind Kesiraju Bob van Luijt
This is my paper · ORCID
classification cs.DBcs.AIcs.IR
keywords FunctionCallingnaturallanguagedatabaseaccesstext-to-SQLtooluseDBGorillabenchmarkquery_databaseschemaExactMatchevaluationsynthetic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that Function Calling, the standard interface for letting a language model invoke external tools, is a practical and generalizable way for LLMs to query databases. Rather than generating SQL, the model fills in a structured query_database tool call with optional search, filter, aggregation, and grouping arguments, which the authors argue separates the conceptual query operators from database-specific syntax. To test this, they build DBGorilla, a synthetic benchmark of 315 natural-language-to-query pairs across five database schemas, and evaluate eight LLMs from five model families. Top models exceed 70% Exact Match, with Claude 3.5 Sonnet reaching 74.3%, suggesting that tool-call generation is competitive with text-to-SQL while being portable across database dialects.

What carries the argument

The load-bearing artifact is the query_database tool definition, a JSON schema with a required collection_name argument restricted to an enumerated list of available collections, and optional search_query, integer/text/boolean property filters, integer/text/boolean aggregations, and groupby_property arguments. This schema unifies access to structured data with semantic search and lets one interface route queries across multiple collections. The DBGorilla dataset is built by a two-stage pipeline: GPT-4o generates synthetic schemas, then a Self-Instruct style procedure with a Reflexion-style validation step produces 63 query operator combinations per schema, totaling 315 queries. Evaluation uses Exact Match on the predicted function arguments as the primary metric, with Abstract Syntax Tree structural scoring and LLM-as-judge preference rankings as secondary views.

What would settle it

Take a set of human-written natural-language database requests over the same five schemas, or over new schemas written by humans, and run the same models; if accuracy drops well below the reported Exact Match scores or the error profile changes materially, such as text filters ceasing to be the hardest component, that would show DBGorilla measures a GPT-4o generation style rather than a generalizable capability.

Watch

Extended reading notes

Core claim

On its own terms, the paper demonstrates that a single JSON tool definition, query_database, can carry search, filtering, aggregation, and grouping operations, and that leading LLMs can fill it out from natural language commands with high fidelity. On the DBGorilla benchmark, Claude 3.5 Sonnet reaches 74.3% Exact Match, GPT-4o mini 73.7%, GPT-4o 71.8%, and Gemini 1.5 Pro 70.2%. The authors report that models are strongest with boolean-property filters and weakest at text-property filters, which they often mistake for search queries. Ablation studies with GPT-4o find that adding rationales, enabling parallel tool calls, splitting the tool per collection, or using structured outputs instead of a vendor Function Calling SDK changes Exact Match by less than about two points. The paper concludes that Function Calling is a promising foundation for bridging human intent and database operations.

Load-bearing premise

The benchmark's synthetic schemas and natural language commands, generated by GPT-4o, faithfully capture the difficulty and variety of real-world database querying, so that Exact Match scores on DBGorilla predict how well the models would perform on real user requests.

Editorial extensions

If this is right

  • Natural language database interfaces can be built by exposing a query_database tool instead of generating SQL, making the same interface work across relational, document, and search databases.
  • The near-parity of GPT-4o mini (73.7%) with Claude 3.5 Sonnet (74.3%) at roughly one-twentieth the cost makes low-cost database agents practical today.
  • A single tool definition can replace separate retrieval and text-to-SQL pipelines in compound AI systems, since it covers both search-style retrieval and structured aggregation.
  • The strong and consistent AST scores of top models (above 0.95) mean that even when a prediction fails Exact Match, it is structurally close, leaving room for lightweight repair rather than full retraining.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because GPT-4o generated both the schemas and the queries, part of the measured accuracy may reflect stylistic familiarity; an independent, human-authored test set is needed to rule out an evaluation artifact.
  • The consistent confusion between text filters and search queries suggests a testable schema redesign: making the filter type explicit, such as 'exact text match' versus 'semantic search', could close much of the observed gap.
  • The cost analysis in the paper (about $8.10 per full benchmark run, roughly $100 per year for monthly updates) implies that continuously tracking LLM database-querying progress is feasible, which would make the benchmark a live leaderboard.
  • Extending DBGorilla with foreign keys, multi-hop queries, and iterative querying would stress-test whether the tool-call interface generalizes beyond the current single-step, single-operator combinations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 1 minor

Summary. The paper proposes a JSON-based tool definition, query_database, for using LLM Function Calling to query a search database, and introduces DBGorilla, a synthetic benchmark with five schemas (three collections, four properties each) and 315 natural-language-to-function-call pairs generated by GPT-4o via Self-Instruct and Reflexion. The authors evaluate eight LLMs from five model families on Exact Match, AST alignment, collection routing, and LLM-as-judge preference rankings, reporting that Claude 3.5 Sonnet achieves the highest Exact Match (74.3%), followed by GPT-4o mini (73.7%), GPT-4o (71.8%), and Gemini 1.5 Pro (70.2%). They also run ablations on rationale arguments, parallel tool calls, per-collection tools, and structured generation, finding little performance change. The paper claims that Function Calling provides an effective and generalizable interface for natural-language database access.

Significance. If the claims are validated, the paper makes a useful practical contribution: a concrete function-calling schema for database queries, an open-source benchmark and evaluation pipeline, and a transparent cost analysis that enables cheap replication. The open-sourced code and the explicit listing of model costs are strengths. However, the significance of the central claim is currently limited by the fully synthetic, GPT-4o-generated benchmark and by the absence of statistical inference, so the 'effective and generalizable' conclusion in the abstract and conclusion is stronger than the evidence supports. The paper is likely to be of interest to practitioners building compound AI systems with database tools, but it does not yet provide a validated general benchmark comparable to human-annotated text-to-SQL sets.

major comments (1)
  1. [Section 4.3 (Preference rankings)] The paper describes the LLM-as-judge results as reflecting 'user preference' and 'readability, clarity, or perceived helpfulness,' but the evaluation uses an LLM judge, not human raters. Statements such as 'Gemini 1.5 Pro, GPT-4o mini and GPT-4o emerge as the most favored models' (Table 5) are not grounded in human judgments. Please relabel the metric as LLM-as-judge preference, or add a human evaluation study to support the interpretation.
minor comments (1)
  1. [Abstract] The abstract uses 'In this report' while the body uses 'this work'; please unify the terminology.

Circularity Check

0 steps flagged · score 2.0 of 10

No reduction-by-construction; the core evaluation is experimental. The only mild concern is a non-load-bearing self-citation and a GPT-4o-generated benchmark used to evaluate GPT-4o-family models.

full rationale

The paper's conclusion that Function Calling is an effective interface is supported by direct measurements (Exact Match, AST, routing accuracy) on the DBGorilla benchmark, not by fitting parameters or by a derived equation. The ground-truth queries are fixed by pre-defined operator combinations and Pydantic structured-output models, and the top Exact Match score is achieved by Claude 3.5 Sonnet (74.3%), not by the model family that generated the dataset, so the result is not forced by construction. The benchmark's schemas and natural-language commands are generated by GPT-4o (Sections 3.2.1-3.2.2), and GPT-4o-family models are among the evaluated systems; this creates a measurement-validity concern (the test set may reward imitation of GPT-4o's phrasing/style), but it is a benchmark-design risk, not a circular derivation. The only self-citation is reference [30] (StructuredRAG, by overlapping authors) in Section 3.2 as one of several structured-generation references; it is not load-bearing because the paper's own Pydantic-based procedure and the external references [28, 29] carry the methodological weight. No self-citation chain or imported uniqueness theorem is used to force the choice of tool schema. Therefore no circular step meets the evidence bar; score 2 reflects the minor self-citation and the same-generator evaluation concern rather than any reduction by construction.

Assumptions & free parameters 2 free parameters · 2 assumptions · 0 invented entities

No free parameters are fitted to data; the only hand-chosen numbers are the scoring weights in the AST and preference-ranking metrics. The main axioms are that the synthetic benchmark distribution matches real-world tasks and that Exact Match is a meaningful correctness measure.

free parameters (2)
  • AST component weights = collection 40%, search/filter/aggregation/groupby 15% each
    Hand-chosen weights in the Abstract Syntax Tree scoring function, which directly affect the reported AST scores and the model rankings based on them.
  • Preference rank weights = 100, 70, 50, 35, 25, 20, 15, 10 (for ranks 1-8)
    Hand-chosen point values in the LLM-as-judge weighted ranking, affecting the preference ranking scores.
assumptions (2)
  • domain assumption Synthetic schemas and natural language queries generated by GPT-4o are representative of real-world database querying tasks.
    Invoked in Section 3.2: the benchmark is created with GPT-4o and assumed to support conclusions about general LLM database querying effectiveness.
  • domain assumption Exact Match of query API calls is a valid proxy for query correctness.
    The paper uses Exact Match as its primary metric (Section 3.3) without validating that exact API equality corresponds to correct query execution results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Querying Databases with Function Calling." pith.science (2026). https://pith.science/paper/XHOZYFU2

@misc{pith2026250200032,
  author       = {Pith},
  title        = {Pith review of: Querying Databases with Function Calling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XHOZYFU2}},
  note         = {Machine review of arXiv:2502.00032}
}
read the original abstract

The capabilities of Large Language Models (LLMs) are rapidly accelerating largely thanks to their integration with external tools. Querying databases is among the most effective of these integrations, enabling LLMs to access private or continually updating data. While Function Calling is the most common method for interfacing external tools to LLMs, its application to database querying as a tool has been underexplored. We propose a tool definition for database querying that unifies accessing data with search queries, filters, or a combination both, as well as transforming results with aggregation and groupby operators. To evaluate its effectiveness, we conduct a study with 8 LLMs spanning 5 model families. We present a novel pipeline adapting the Gorilla LLM framework to create synthetic database schemas and queries. We primarily evaluate the models with the Exact Match of predicted and ground truth query APIs. Among the models tested, Claude 3.5 Sonnet achieves the highest performance with an Exact Match score of 74.3%, followed by GPT-4o mini at 73.7%, and GPT-4o at 71.8%. We further breakdown these results per API component utilized and across synthetic use cases. We find that LLMs are highly effective at utilizing operators on boolean properties, but struggle with text property filters. Across use cases we find robust results with the higher performing models such as GPT-4o, but significant performance variance across use cases from lower performing models. We additionally conduct ablation studies exploring the impact of parallel tool calling, adding a rationale as an argument of the tool call, using a separate tool per database collection, and tool calling with structured outputs. Our findings demonstrate the effectiveness of enabling LLMs to query databases with Function Calling. We have open-sourced our experimental code and results at github.com/weaviate/gorilla.

Figures

Figures reproduced from arXiv: 2502.00032 by the authors.

Figure 1
Figure 1. DBGorilla Leaderboard results (last updated January 1st, 2025). The Exact Match and AST Score columns [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. An illustration of a natural language command, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Examples of queries in the BIRD Text-to-SQL benchmark [ [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: An illustration of the Function Calling loop. Beginning with the user’s input prompt, the LLM then enters a [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Radar plots highlighting how well each model tested can access particular Search Database API components. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Visualization tool for manually inspecting synthetic query quality and results. [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: An illustration of natural language commands translated to Function Calling arguments for our proposed tool [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. OpenClaw and Ollama in Agentic AI: Toward Fully Autonomous and Scalable AI Agent Systems

    cs.AI 2026-04 reject novelty 3.0 of 10

    A review-plus-demo claiming agentic capabilities emerge from system integration, backed by a 15-task benchmark whose C1→C3 performance gap is largely built into the test design.

Reference graph

Works this paper leans on

43 extracted references · 13 canonical work pages · cited by 1 Pith paper

  1. [1]

    The shift from models to compound ai systems

    Matei Zaharia, Omar Khattab, Lingjiao Chen, Jared Quincy Davis, Heather Miller, Chris Potts, James Zou, Michael Carbin, Jonathan Frankle, Naveen Rao, and Ali Ghodsi. The shift from models to compound ai systems. https://bair.berkeley.edu/blog/2024/02/18/compound-ai-systems/, 2024

    2024

  2. [2]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2023

    arXiv 2023

  3. [3]

    Patil, Tianjun Zhang, Xin Wang, and Joseph E

    Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334, 2023

    arXiv 2023

  4. [4]

    Function calling in the chat completions api

    OpenAI. Function calling in the chat completions api. https://platform.openai.com/docs/guides/ function-calling. Accessed: January 3, 2025

    work page 2025

  5. [5]

    Spider 2.0: Evaluating language models on real-world enterprise text-to-sql workflows

    Fangyu Lei, Yujie Zhu, Wanjun Zhu, Qian Yin, Yicheng Yin, Jiawei Yin, Yusen Zhuang, Bowen Qin, Victor Zhong, Xuandong Yin, et al. Spider 2.0: Evaluating language models on real-world enterprise text-to-sql workflows. arXiv preprint arXiv:2411.07763, 2024

    arXiv 2024

  6. [6]

    Ioannidis, Karthik Subbian, James Zou, and Jure Leskovec

    Shirley Wu, Shiyu Zhao, Michihiro Yasunaga, Kexin Huang, Kaidi Cao, Qian Huang, Vassilis N. Ioannidis, Karthik Subbian, James Zou, and Jure Leskovec. Stark: Benchmarking llm retrieval on textual and relational knowledge bases. arXiv preprint arXiv:2404.13207, 2024. 12

    arXiv 2024

  7. [7]

    https://www.w3.org/TR/sparql11-query/, 2013

    Sparql query language. https://www.w3.org/TR/sparql11-query/, 2013

    work page 2013

  8. [8]

    Semantic operators: A declarative model for rich, ai-based analytics over text data

    Liana Patel, Siddharth Jha, Parth Asawa, Melissa Pan, Carlos Guestrin, and Matei Zaharia. Semantic operators: A declarative model for rich, ai-based analytics over text data. arXiv preprint arXiv:2407.11418, 2024

    arXiv 2024

Show all 43 references
  1. [9]

    Gonzalez, Carlos Guestrin, and Matei Zaharia

    Asim Biswal, Liana Patel, Siddarth Jha, Amog Kamsetty, Shu Liu, Joseph E. Gonzalez, Carlos Guestrin, and Matei Zaharia. Text2sql is not enough: Unifying ai and databases with tag. arXiv preprint arXiv:2408.14717, 2024

    2024 arXiv

  2. [10]

    Semnani, Chen Jie Yu, and Monica S

    Shicheng Liu, Jialiang Xu, Wesley Tjangnaka, Sina J. Semnani, Chen Jie Yu, and Monica S. Lam. Suql: Conversa- tional search over structured and unstructured data with large language models. arXiv preprint arXiv:2311.09818, 2024

    2024 arXiv

  3. [11]

    Patil, Ion Stoica, and Joseph E

    Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. Berkeley function calling leaderboard. https://gorilla.cs.berkeley.edu/blogs/8_berkeley_ function_calling_leaderboard.html, 2024

    2024

  4. [12]

    Code generation with alphacodium: From prompt engineering to flow engineering

    Tal Ridnik, Dedy Kredo, and Itamar Friedman. Code generation with alphacodium: From prompt engineering to flow engineering. arXiv preprint arXiv:2401.08500, 2024

    2024 arXiv

  5. [13]

    Retrieval-augmented generation for knowledge-intensive nlp tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. In H. Larochelle, M. ...

    2020

  6. [14]

    Webgpt: Browser-assisted question-answering with human feedback

    Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webg...

    2022 arXiv

  7. [15]

    Agentic information retrieval

    Weinan Zhang, Junwei Liao, Ning Li, and Kounianhua Du. Agentic information retrieval. arXiv preprint arXiv:2410.09713, 2024

    2024 arXiv

  8. [16]

    Next- generation database interfaces: A survey of llm-based text-to-sql

    Zijin Hong, Zheng Yuan, Qinggang Zhang, Hao Chen, Junnan Dong, Feiran Huang, and Xiao Huang. Next- generation database interfaces: A survey of llm-based text-to-sql. arXiv preprint arXiv:2406.08426, 2024

    2024

  9. [17]

    Seq2sql: Generating structured queries from natural language using reinforcement learning

    Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint arXiv:1709.00103, 2017

    2017 arXiv

  10. [18]

    Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task

    Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceed...

    2018

  11. [19]

    Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls

    Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, et al. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. Advances in Neural Information Processing Systems ,...

    2024

  12. [20]

    Lakehouse: A new generation of open platforms that unify data warehousing and advanced analytics

    Matei Zaharia, Ali Ghodsi, Reynold Xin, and Michael Armbrust. Lakehouse: A new generation of open platforms that unify data warehousing and advanced analytics. In 11th Conference on Innovative Data Systems Research, CIDR 2021, Virtual Event, January 11-15, 2021, Online Proceed...

    2021

  13. [21]

    Bringing semantic knowledge graph technology to your data

    Bob van Luijt and Micha Verhagen. Bringing semantic knowledge graph technology to your data. IEEE Software, 37(2):89–94, 2020

    2020

  14. [22]

    Database gyms

    Wan Shen Lim, Matthew Butrovich, William Zhang, Andrew Crotty, Lin Ma, Peijing Xu, Johannes Gehrke, and Andrew Pavlo. Database gyms. Conference on Innovative Data Systems Research

  15. [23]

    Gpt-4 technical report

    OpenAI and Josh Achiam et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2024

    2024 arXiv

  16. [24]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Gemini Team Google. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  17. [25]

    https://www.anthropic.com/news/claude-3-5-sonnet , 2024

    Claude 3.5 sonnet. https://www.anthropic.com/news/claude-3-5-sonnet , 2024

    2024

  18. [26]

    https://docs.cohere.com/v2/docs/command-r, 2024

    The command r model (details and applications). https://docs.cohere.com/v2/docs/command-r, 2024. 13

    2024

  19. [27]

    Llama 3 model card, Accessed July 2024

    AI@Meta. Llama 3 model card, Accessed July 2024

    2024

  20. [28]

    Willard and Rémi Louf

    Brandon T. Willard and Rémi Louf. Efficient guided generation for large language models. arXiv preprint arXiv:2307.09702, 2023

    2023 arXiv

  21. [29]

    Let me speak freely? a study on the impact of format restrictions on performance of large language models

    Zhi Rui Tam, Cheng-Kuang Wu, Yi-Lin Tsai, Chieh-Yen Lin, Hung yi Lee, and Yun-Nung Chen. Let me speak freely? a study on the impact of format restrictions on performance of large language models. arXiv preprint arXiv:2408.02442, 2024

    2024 arXiv

  22. [30]

    Structuredrag: Json response formatting with large language models

    Connor Shorten, Charles Pierse, Thomas Benjamin Smith, Erika Cardenas, Akanksha Sharma, John Trengrove, and Bob van Luijt. Structuredrag: Json response formatting with large language models. arXiv preprint arXiv:2408.11061, 2024

    2024 arXiv

  23. [31]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st A...

    2023

  24. [33]

    Reflexion: Language agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. arXiv preprint arXiv:2303.11366, 2023

    2023 arXiv

  25. [34]

    Dspy assertions: Computational constraints for self-refining language model pipelines

    Arnav Singhvi, Manish Shetty, Shangyin Tan, Christopher Potts, Koushik Sen, Matei Zaharia, and Omar Khat- tab. Dspy assertions: Computational constraints for self-refining language model pipelines. arXiv preprint arXiv:2312.13382, 2024

    2024 arXiv

  26. [35]

    Shreya Shankar, Haotian Li, Parth Asawa, Madelon Hulsebos, Yiming Lin, J. D. Zamfirescu-Pereira, Harrison Chase, Will Fu-Hinthorn, Aditya G. Parameswaran, and Eugene Wu. Spade: Synthesizing data quality assertions for large language model pipelines. arXiv preprint arXiv:2401.0...

    2024 arXiv

  27. [36]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024

    2024

  28. [37]

    Anchored preference optimization and contrastive revisions: Addressing underspecification in alignment

    Karel D’Oosterlinck, Winnie Xu, Chris Develder, Thomas Demeester, Amanpreet Singh, Christopher Potts, Douwe Kiela, and Shikib Mehri. Anchored preference optimization and contrastive revisions: Addressing underspecification in alignment. arXiv preprint arXiv:2408.06266, 2024

    2024 arXiv

  29. [38]

    Patil, Tianjun Zhang, Vivian Fang, Noppapon C., Roy Huang, Aaron Hao, Martin Casado, Joseph E

    Shishir G. Patil, Tianjun Zhang, Vivian Fang, Noppapon C., Roy Huang, Aaron Hao, Martin Casado, Joseph E. Gonzalez, Raluca Ada Popa, and Ion Stoica. GoEX: Perspectives and designs towards a runtime for autonomous llm applications. arXiv preprint arXiv:2404.06921, 2024

    2024 arXiv

  30. [39]

    Networks of networks: Complexity class principles applied to compound ai systems design

    Jared Quincy Davis, Boris Hanin, Lingjiao Chen, Peter Bailis, Ion Stoica, and Matei Zaharia. Networks of networks: Complexity class principles applied to compound ai systems design. arXiv preprint arXiv:2407.16831, 2024

    2024 arXiv

  31. [40]

    Patil, Lingjiao Chen, Wei-Lin Chiang, and Jared Q

    Ion Stoica, Matei Zaharia, Joseph Gonzalez, Ken Goldberg, Koushik Sen, Hao Zhang, Anastasios Angelopoulos, Shishir G. Patil, Lingjiao Chen, Wei-Lin Chiang, and Jared Q. Davis. Specifications: The missing link to making the development of llm systems an engineering discipline. ...

    2024 arXiv

  32. [41]

    Demonstrate-search-predict: Composing retrieval and language models for knowledge-intensive NLP

    Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. Demonstrate-search-predict: Composing retrieval and language models for knowledge-intensive NLP. arXiv preprint arXiv:2212.14024, 2022

    2022 arXiv

  33. [42]

    Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. Dspy: Compiling declarative language model calls into self-impro...

    arXiv

  34. [43]

    Optimizing instructions and demonstrations for multi-stage language model programs

    Krista Opsahl-Ong, Michael J Ryan, Josh Purtell, David Broman, Christopher Potts, Matei Zaharia, and Omar Khattab. Optimizing instructions and demonstrations for multi-stage language model programs. arXiv preprint arXiv:2406.11695, 2024

    2024 arXiv

  35. [44]

    type": " function

    Shirley Wu, Shiyu Zhao, Qian Huang, Kexin Huang, Michihiro Yasunaga, Kaidi Cao, Vassilis N. Ioannidis, 14 Karthik Subbian, Jure Leskovec, and James Zou. Avatar: Optimizing llm agents for tool usage via contrastive reasoning. arXiv preprint arXiv:2406.11200, 2024. 15 A Primary ...

    2024 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.