Pith. sign in

REVIEW 4 major objections 4 minor 56 references

OPTI-Q proposes that answering a question with several LLMs should be a database-style planning problem: pick a per-question execution DAG from measured statistics before any model runs, then execute it under the user's budget.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 10:57 UTC pith:5QXRITSO

load-bearing objection A well-specified planning system with a genuinely useful statistics catalog, but the SimpleQA results rest on an embedding-similarity proxy that undermines the factual-quality claim. the 4 major comments →

arxiv 2607.22621 v1 pith:5QXRITSO submitted 2026-06-17 cs.AI

Opti-Q: A Constraint-Based Optimization Framework for Multi-LLM Question Planning

Aamir Hamid , Bharg Barot , Satvik Racharla , Tim Finin , Primal Pappachan , Roberto Yus This is my paper
classification cs.AI
keywords multi-LLM orchestrationquery optimizationcost-based planningPareto frontierquality-resource trade-offquestion answeringstatistics catalog
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

OPTI-Q argues that orchestration of multiple LLMs for a single question should be treated as query optimization: the system constructs a DAG of model calls—sequential chains, parallel runs, and output blending—and estimates each candidate plan's answer quality, dollar cost, latency, and energy from a statistics catalog before executing anything. It then searches for Pareto-optimal plans under the user's budget and selects one by user weights. On sampled questions from MMLU-Pro and SimpleQA, the paper reports roughly 58% and 41% higher average QoA than budget-aware routing and ensemble baselines at comparable per-question cost. The central bet is that the best model combination and execution structure are question-dependent, and that historical statistics predict them better than greedy routing rules.

Core claim

The discovery OPTI-Q tries to establish is that the cost-based optimization discipline used for SQL queries transfers to composing LLM calls. Model invocations are treated as physical operators with measured quality and resource profiles, and a planner synthesizes a question-specific execution DAG before execution. Using a statistics catalog (PERFDB) built from benchmarks and execution traces, OPTI-Q estimates token volumes, quality, and resources for candidate plans, then runs Pareto-frontier search—exact dynamic programming for small spaces, evolutionary search for larger ones—to pick a plan under user budgets. The empirical claim is that this beats routing, cascading, and ensembling basel

What carries the argument

PERFDB, a topic-conditioned statistics catalog keyed by execution context, plus a compact canonical DAG encoding of plans (an upper-triangular adjacency bit vector with model assignments). PERFDB stores QoA and resource statistics for single models and composed subplans; the planner composes them into pre-execution estimates using a multiplicative relative-effect rule for sequential steps and an averaging blend rule for parallel-plus-blend nodes, with fallbacks for missing entries. The cost model is tokenizer-aware, applying each model's own tokenizer, expected output lengths from PERFDB, and fixed-plus-variable pricing, with parallel latency taken as the slowest branch. This machinery lets

Load-bearing premise

The headline margins are measured against author re-implementations of the published baselines, adapted to OPTI-Q's per-question budget scheme; if those re-implementations are weaker than the original systems, the advantage reflects the comparison setup rather than the planner.

What would settle it

Re-run the comparison using the original, unmodified published baseline systems under the same per-question budgets and identical measurement of QoA, cost, latency, and energy on the 200 sampled MMLU-Pro and SimpleQA questions; if the roughly 58% and 41% QoA advantages at budget level 3 do not persist, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Overruns stay rare under cost and latency budgets: 88–97% of executed plans adhere to the budget, with overruns driven mainly by financial cost.
  • Plan complexity pays off most on classification-style reasoning: increasing the operation limit from k=1 to k=5 yields +54.7% QoA on MMLU-Pro versus +23.6% on SimpleQA, and at k=5 parallel and hybrid plans account for roughly 82% of selected plans.
  • Diversity saturates: the largest marginal QoA gain appears when moving from 2 to 3 distinct models (ΔQoA=0.038), while moving from 4 to 5 adds only 0.008.
  • OPTI-Q beats strong commercial API models on SimpleQA on cost-adjusted quality, with one leading API model 95.2x more expensive for modestly better QoA.
  • Richer PERFDB coverage improves planning: MMLU-Pro QoA rises from 0.40 at cold start to 0.67 at Level 4 coverage, and estimation error drops sharply across cost, energy, latency, and QoA.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the headline margins are real rather than an artifact of the baseline re-implementations, the plan-before-execute design likely transfers beyond QA to RAG and agentic workflows, where retrieval, reranking, verification, and tool calls are also operators with measurable cost and quality profiles.
  • The planning overhead (about 21 seconds at k=5 with the evolutionary backend) suggests per-question planning is best suited to offline or asynchronous use; caching plans keyed by topic or by similar questions, or moving to cheaper heuristic planners for interactive settings, would make the approach practical at low latency.
  • Even with full PERFDB coverage, QoA estimates remain imperfect (MAE around 0.27–0.34), so the planner's advantage likely comes from preserving relative ranking among candidate plans rather than from accurate absolute quality prediction—a property that should be tested explicitly as the catalog grows.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents OPTI-Q, a plan-before-execute optimizer for multi-LLM question answering. It encodes candidate execution workflows as DAGs of LLM invocations with sequential, parallel, and blending operators; estimates QoA, financial cost, latency, and energy from a PERFDB statistics catalog; and searches the plan space using NSGA-II, dynamic programming, or hill climbing under user-specified budgets. The authors claim roughly 58% and 41% average QoA improvements over budget-aware baselines on MMLU-Pro and SimpleQA, and argue that database-style planning yields better quality-resource trade-offs. The main text includes the formal problem statement, the DAG encoding, cost/QoA estimators, and an evaluation on 200 sampled questions with five locally hosted models.

Significance. The framework is well structured and the cost/benefit formulation is a useful contribution. The strengths include a precise DAG encoding with canonicalization, token-aware cost estimators, a clean separation between PERFDB statistics, planners, and execution, and the release of code/results. The held-out configuration set is good practice. However, the headline empirical claims are not yet established: the SimpleQA quality metric is a cosine-similarity proxy rather than the benchmark's official factual-correctness metric; the evaluation is based on 200 questions with no reported confidence intervals; and all comparative numbers are against reimplemented baselines that are not validated. If the SimpleQA results were re-scored on the official metric and the MMLU-Pro results hold, this would be a solid systems contribution.

major comments (4)
  1. [§V, §VI-A(b), §VI-D] For SimpleQA, QoA is defined as cosine similarity between all-MiniLM-L6-v2 embeddings of the predicted and reference answers, not the benchmark's official exact-match/contains correctness. Because PERFDB stores and OPTI-Q optimizes this embedding-similarity score, the headline 41% SimpleQA gain and Table III's comparisons against Claude/GPT do not establish improved factual QA. Re-score all SimpleQA results on the official metric, or explicitly relabel the claim as 'semantic-similarity QoA' and remove SimpleQA from the factual-accuracy headline.
  2. [§VI] The entire comparative evaluation uses 100 questions per benchmark (10 questions each from 10 sampled types) executed five times. The text says 95% confidence intervals are reported, but Figures 3–5 and Tables I–III report only point estimates, with no intervals or significance tests. Given the small sample, the '≈58%' and '≈41%' margins cannot be assessed statistically. Report CIs and ideally bootstrap or a larger testbed.
  3. [§VI, Baselines and scope] Baselines are author re-implementations: FrugalGPT*, ThriftLLM*, LLM-Ensemble*, LLM-Blender*, with FrugalGPT* adapted to a per-query budget and LLM-Ensemble* given a greedy budgeted variant. No evidence is provided that these reimplementations reproduce the published methods' behavior. The central comparisons in Figure 3 are therefore against a possibly weaker reference point. Provide validation against published numbers or a component-level ablation, and state explicitly which baseline is the 'strongest budget-aware baseline' used for the 58%/41% figures.
  4. [Abstract, §VI-A] The claimed improvement magnitudes are not consistent with the body's numbers. For SimpleQA at b=3, the text reports OPTI-Q QoA ≈0.56–0.61 versus FrugalGPT* ≈0.46, which is a ~22–33% relative gain, not ~41%; the 41% figure appears to correspond to b=5 (0.65 vs 0.46). For MMLU-Pro, the b=3 gain (≈77% vs ≈45–47% baselines) is ~64%, not ~58%. The abstract and introduction must be reconciled with the actual reported operating point.
minor comments (4)
  1. [§IV-A] The canonicalization procedure (lexicographically maximal bit vector over all topological orderings) needs a precise algorithm and complexity analysis; the paper only states O(k^2) space. The extended version should contain this and the full enumeration method.
  2. [§IV-B, QoA Estimation] The sequential and blend update rules are heuristic; their free parameters and fallback choices (e.g., neutral factor 1, average-relative change) deserve a sensitivity analysis, especially since the robustness experiment only reports overall MAE, not per-component calibration.
  3. [§VI-B] The cumulative PERFDB trace counts read as '135,985,1,735,2,360'; likely missing separators. Also, the figure captions reference panels (4a/4b) but the text sometimes cites 'Figures 4a and 4b' without consistent panel labels.
  4. [General] For a journal version, the main text should not delegate so much essential material to an external 'extended version' ([39]) — including the full planner comparison, complete QoA lookup procedure, and additional sensitivity analyses. These are central to reproducibility.

Circularity Check

0 steps flagged

No circularity found: PERFDB estimates are empirical statistics, and the headline result is evaluated on held-out questions disjoint from the training traces; remaining concerns are evaluation-validity issues, not circularity.

full rationale

I found no circular step in OPTI-Q's derivation chain. The planning-time QoA, cost, latency, and energy values are explicitly empirical statistics stored in PERFDB and populated from benchmarks and execution traces; the update rules in Section IV-B (sequential refinement and parallel-blend scaling) are transparent aggregations of stored reference values, not derivations of the target result from its own outcome. The paper states the estimation goal is 'not perfect calibration, but a conservative estimate that preserves relative differences,' which is a statistical estimator, not a first-principles prediction. The empirical claim is evaluated on held-out questions: the configuration set is 'strictly disjoint from the test set' (Section V), and the PERFDB coverage experiments use traces from 'disjoint training questions' to 'avoid data leakage' (Section VI-B). Thus the 58%/41% headline margins do not reduce to the fitted statistics by construction. The SimpleQA cosine-similarity metric and the re-implementation of baselines are validity/comparability concerns, not circularity under the rubric. The only self-citation is to the authors' extended version [39] for supplementary derivations and algorithm details; it is not load-bearing for the central claim and involves no imported uniqueness theorem or ansatz. The derivation is self-contained for the purpose of the circularity analysis.

Axiom & Free-Parameter Ledger

7 free parameters · 6 axioms · 1 invented entities

The central claim rests on empirical statistics (PERFDB), hand-chosen estimation rules, and a small set of user-facing anchors. None of these are hidden: the paper reports them. The most consequential choices are the budget anchors and the compositional QoA update rules, both of which affect which plans the optimizer selects.

free parameters (7)
  • Financial budget anchor = 1.2e-5 USD/question
    Defines F_max(b)=b*1.2e-5; is the average observed per-question cost across the five local models. Used to set all budget levels.
  • Latency budget anchor = 19.47 s/question
    Defines L_max(b)=b*19.47; is the mean observed latency across models.
  • PERFDB fuzzy-match threshold = 0.75
    Fixed threshold for accepting fuzzy structural matches during QoA lookup; hand-set.
  • DP pruning delta = 0.05
    Retain a child plan only if QoA(child) >= QoA(parent)+0.05; hand-set in Section IV-C.
  • NSGA-II hyperparameters = P=200, G=200, mutation rates (0.3, 0.1, 0.3)
    Grid-searched on a held-out 200-question stratified set; chosen by best mean scalarized score in the converged population.
  • Sequential QoA update rule = QoA_next = clip(QoA_curr * QoA(Seq(Li,Lj)) / max(eps, QoA(Lj)))
    Multiplicative relative-effect model; a modeling choice justified qualitatively by observed scaling rather than fitted to data.
  • Blend QoA update rule = QoA_new = clip(QoA_ref_blend * (1 + (1/k) sum((QoA_new_Ls - QoA_ref_Ls)/max(eps, QoA_ref_Ls))))
    Average-relative-change model; deliberately conservative and chosen by the authors to avoid over-crediting blends.
axioms (6)
  • domain assumption PERFDB topic-conditioned statistics transfer to new questions with the same inferred topic
    Section IV-B QoA Estimation: topic-conditioned lookup and aggregation assume historical traces are representative of future questions.
  • domain assumption Estimated plan QoA preserves relative differences between candidate plans
    Section IV-B: 'The goal is not perfect calibration, but a conservative estimate that preserves relative differences between candidate plans, which is sufficient for Pareto-based search.'
  • domain assumption Latency and energy are linear in token counts; financial cost is fixed-plus-variable per token
    Section IV-B Financial/Energy/Latency estimation: simple linear models for real serving behavior.
  • domain assumption Semantic embedding cosine similarity is a valid QoA measure for open-ended QA
    Section V QoA Computation: SimpleQA is a factuality benchmark, but correctness is replaced by all-MiniLM-L6-v2 cosine similarity.
  • standard math Plan space is finite and optimizing over it is NP-hard
    Section IV-C: |Pi|=|L|^k * 2^{k(k-1)/2} and NP-hardness cited to [32].
  • ad hoc to paper Admissible plans must satisfy the blending constraint: each parallel operation is immediately followed by exactly one blending operation
    Section III-A Plan Model: this restricts the search space and may exclude valid high-quality workflows.
invented entities (1)
  • PERFDB independent evidence
    purpose: Central statistics catalog keyed by execution context; supplies pre-execution estimates of QoA, cost, latency, and energy for atomic and composed operations.
    Software component with released code/artifacts; its estimates are directly audited in the Section VI-B coverage experiments, providing a falsifiable handle.

pith-pipeline@v1.3.0-alltime-deepseek · 22490 in / 15082 out tokens · 134850 ms · 2026-08-02T10:57:19.399115+00:00 · methodology

0 comments
read the original abstract

While large language models (LLMs) enable strong question answering (QA), budgeted deployment is complicated by nondeterminism and heterogeneous resource profiles (cost, latency, and energy). We present OPTI-Q, a database-inspired, cost-based optimizer that implements a plan-before-execute paradigm for multi-LLM orchestration. OPTI-Q models LLM invocations as physical operators in an execution DAG and, for each question, searches for plans that optimize answer quality (QoA) while trading off financial cost, latency, and energy under user-specified resource constraints. Plans can include sequential operators that pass intermediate answers as context and parallel/blend operators that run models concurrently and merge their outputs. To search this space without executing each candidate plan, OPTI-Q uses PERFDB, a statistics catalog populated and refreshed from benchmarks and execution traces, to estimate the QoA and resource costs of both individual operators and composed subplans. Using these estimates, OPTI-Q performs Pareto-frontier search and selects a final plan based on user preferences. On MMLU-Pro and SimpleQA under user-specified budgets, OPTI-Q improves average QoA by ~58% and ~41% over baselines at comparable cost, demonstrating that database-style planning yields better quality-resource trade-offs for multi-LLM QA.

Figures

Figures reproduced from arXiv: 2607.22621 by Aamir Hamid, Bharg Barot, Primal Pappachan, Roberto Yus, Satvik Racharla, Tim Finin.

Figure 1
Figure 1. Figure 1: Comparison of candidate multi-LLM execution plans [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: shows an example plan with its encoding. The DAG topology over k operations is encoded by the upper￾triangular adjacency matrix M ∈ {0, 1} k×k , with Mij = 1 for (1 ≤ i < j ≤ k) iff there is a data-flow edge from Li to Lj . For computational efficiency, we flatten M into a k(k − 1)/2-bit vector B = (b12, b13, . . . , b(k−1)k), where each bit bij ∈ {0, 1} indicates whether Li feeds its output to Lj . Blendi… view at source ↗
Figure 3
Figure 3. Figure 3: QoA versus budget across two benchmarks: (a) [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Actual vs. predicted plan-level cost, energy, latency, [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: QoA vs. budget (budget-adherent plans only). D. Quality–Cost Trade-offs Against Commercial API Models We evaluate whether OPTI-Q remains useful when com￾pared against high-capability state-of-the-art commercial mod￾els accessed through their respective provider APIs (i.e., Claude Opus 4.6, GPT 5.4, and Gemini 3.5 Flash).11 Each commercial model is invoked five times per question using the same benchmark pr… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

56 extracted references · 24 linked inside Pith

  1. [1]

    Introducing gpt-5,

    OpenAI, “Introducing gpt-5,” https://openai.com/index/introducing-gpt -5/, 2025, accessed: 2026-02-20

    2025

  2. [2]

    The llama 3 herd of models,

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughanet al., “The llama 3 herd of models,”arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  3. [3]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,

    D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Biet al., “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,”arXiv preprint arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  4. [4]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” 2019. [Online]. Available: https://arxiv.org/abs/1810.04805

    Pith/arXiv arXiv 2019

  5. [5]

    Reading wikipedia to answer open-domain questions,

    D. Chen, A. Fisch, J. Weston, and A. Bordes, “Reading wikipedia to answer open-domain questions,” 2017. [Online]. Available: https: //arxiv.org/abs/1704.00051

    Pith/arXiv arXiv 2017

  6. [6]

    Retrieval-augmented generation for knowledge-intensive nlp tasks,

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K ¨uttler, M. Lewis, W. tau Yih, T. Rockt ¨aschel, S. Riedel, and D. Kiela, “Retrieval-augmented generation for knowledge-intensive nlp tasks,” 2021. [Online]. Available: https://arxiv.org/abs/2005.11401

    Pith/arXiv arXiv 2021

  7. [7]

    Dense passage retrieval for open-domain question answering,

    V . Karpukhin, B. O ˘guz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W. tau Yih, “Dense passage retrieval for open-domain question answering,” 2020. [Online]. Available: https://arxiv.org/abs/2004.04906

    Pith/arXiv arXiv 2020

  8. [8]

    Neural approaches to conversational ai: Question answering, task-oriented dialogues and social chatbots,

    J. Gao, M. Galley, and L. Li, “Neural approaches to conversational ai: Question answering, task-oriented dialogues and social chatbots,” 2019

    2019

  9. [9]

    Establishing and maintaining long-term human-computer relationships,

    T. W. Bickmore and R. W. Picard, “Establishing and maintaining long-term human-computer relationships,”ACM Trans. Comput. Hum. Interact., vol. 12, pp. 293–327, 2005. [Online]. Available: https://api.semanticscholar.org/CorpusID:946518

    2005

  10. [10]

    Genaipabench: A benchmark for generative ai-based privacy assistants,

    A. Hamid, H. R. Samidi, T. Finin, P. Pappachan, and R. Yus, “Genaipabench: A benchmark for generative ai-based privacy assistants,” arXiv preprint arXiv:2309.05138, vol. n/a, no. n/a, p. n/a, 2023

    Pith/arXiv arXiv 2023

  11. [11]

    The rising costs of training frontier ai models,

    B. Cottier, R. Rahman, L. Fattorini, N. Maslej, T. Besiroglu, and D. Owen, “The rising costs of training frontier ai models,”arXiv preprint arXiv:2405.21015, vol. n/a, no. n/a, p. n/a, 2024

    Pith/arXiv arXiv 2024

  12. [12]

    Energy and policy considerations for deep learning in NLP,

    E. Strubell, A. Ganesh, and A. McCallum, “Energy and policy considerations for deep learning in NLP,” inProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, A. Korhonen, D. Traum, and L. M `arquez, Eds. Florence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 3645–3650. [Online]. Available: https://acl...

    2019

  13. [13]

    The carbon footprint of machine learning training will plateau, then shrink,

    D. Patterson, J. Gonzalez, U. H ¨olzle, Q. Le, C. Liang, L.-M. Munguia, D. Rothchild, D. So, M. Texier, and J. Dean, “The carbon footprint of machine learning training will plateau, then shrink,” 2022. [Online]. Available: https://arxiv.org/abs/2204.05149

    Pith/arXiv arXiv 2022

  14. [14]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhouet al., “Chain-of-thought prompting elicits reasoning in large language models,”Advances in neural information processing systems, vol. 35, pp. 24 824–24 837, 2022

    2022

  15. [15]

    Measuring short-form factuality in large language models,

    J. Wei, N. Karina, H. W. Chung, Y . J. Jiao, S. Papay, A. Glaese, J. Schulman, and W. Fedus, “Measuring short-form factuality in large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2411 .04368

    2024

  16. [16]

    When one llm drools, multi-llm collaboration rules,

    S. Feng, W. Ding, A. Liu, Z. Wang, W. Shi, Y . Wang, Z. Shen, X. Han, H. Lang, C.-Y . Lee, T. Pfister, Y . Choi, and Y . Tsvetkov, “When one llm drools, multi-llm collaboration rules,” 2025. [Online]. Available: https://arxiv.org/abs/2502.04506

    Pith/arXiv arXiv 2025

  17. [17]

    Ensemble methods in machine learning,

    T. G. Dietterich, “Ensemble methods in machine learning,” inProceed- ings of the First International Workshop on Multiple Classifier Systems, ser. MCS ’00. Berlin, Heidelberg: Springer-Verlag, 2000, p. 1–15

    2000

  18. [18]

    Prompt design and engineering: Introduction and advanced methods,

    X. Amatriain, “Prompt design and engineering: Introduction and advanced methods,” 2024. [Online]. Available: https://arxiv.org/abs/24 01.14423

    2024

  19. [19]

    Reducing hallucinations in large language models: A consensus voting approach using mixture of experts,

    S. Suzuoki and K. Hatano, “Reducing hallucinations in large language models: A consensus voting approach using mixture of experts,” 2024

    2024

  20. [20]

    Uncertainty-aware fusion: An ensemble framework for mitigating hallucinations in large language models,

    P. Dey, S. Merugu, and S. Kaveri, “Uncertainty-aware fusion: An ensemble framework for mitigating hallucinations in large language models,”arXiv preprint arXiv:2503.05757, vol. n/a, no. n/a, p. n/a, 2025

    Pith/arXiv arXiv 2025

  21. [21]

    Llm-blender: Ensembling large language models with pairwise ranking and generative fusion,

    D. Jiang, X. Ren, and B. Y . Lin, “Llm-blender: Ensembling large language models with pairwise ranking and generative fusion,” 2023. [Online]. Available: https://arxiv.org/abs/2306.02561

    Pith/arXiv arXiv 2023

  22. [22]

    [Online]

    LangChain,LangChain Documentation, 2024, accessed: 2024-02-05. [Online]. Available: https://python.langchain.com/docs/introduction/

    2024

  23. [23]

    Palimpzest: Optimizing AI-powered analytics with declarative query processing,

    C. Liu, M. Russo, M. Cafarella, L. Cao, P. B. Chen, Z. Chen, M. Franklin, T. Kraska, S. Madden, R. Shahout, and G. Vitagliano, “Palimpzest: Optimizing AI-powered analytics with declarative query processing,” inProceedings of the Conference on Innovative Database Research (CIDR), 2025

    2025

  24. [24]

    Bao: Making learned query optimization practical,

    R. Marcus, P. Negi, H. Mao, N. Tatbul, M. Alizadeh, and T. Kraska, “Bao: Making learned query optimization practical,” in Proceedings of the 2021 International Conference on Management of Data, ser. SIGMOD ’21. New York, NY , USA: Association for Computing Machinery, 2021, p. 1275–1288. [Online]. Available: https://doi.org/10.1145/3448016.3452838

    arXiv 2021

  25. [25]

    Cardinality estimation in dbms: a comprehensive benchmark evaluation,

    Y . Han, Z. Wu, P. Wu, R. Zhu, J. Yang, L. W. Tan, K. Zeng, G. Cong, Y . Qin, A. Pfadler, Z. Qian, J. Zhou, J. Li, and B. Cui, “Cardinality estimation in dbms: a comprehensive benchmark evaluation,”Proc. VLDB Endow., vol. 15, no. 4, p. 752–765, Dec. 2021. [Online]. Available: https://doi.org/10.14778/3503585.3503586

    arXiv 2021

  26. [26]

    A query opti- mization method utilizing large language models,

    Z. Yao, H. Li, J. Zhang, C. Li, and H. Chen, “A query opti- mization method utilizing large language models,”arXiv preprint arXiv:2503.06902, 2025

    Pith/arXiv arXiv 2025

  27. [27]

    Lero: A learning-to-rank query optimizer,

    R. Zhu, W. Chen, B. Ding, X. Chen, A. Pfadler, Z. Wu, and J. Zhou, “Lero: A learning-to-rank query optimizer,”Proc. VLDB Endow., vol. 16, no. 6, p. 1466–1479, Feb. 2023. [Online]. Available: https://doi.org/10.14778/3583140.3583160

    arXiv 2023

  28. [28]

    Dspy: Compiling declarative language model calls into self-improving pipelines,

    O. Khattab, A. Singhvi, P. Maheshwari, Z. Zhang, K. Santhanam, S. Vardhamanan, S. Haq, A. Sharma, T. T. Joshi, H. Moazam, H. Miller, M. Zaharia, and C. Potts, “Dspy: Compiling declarative language model calls into self-improving pipelines,” 2024

    2024

  29. [29]

    Frugalgpt: How to use large language models while reducing cost and improving performance,

    L. Chen, M. Zaharia, and J. Zou, “Frugalgpt: How to use large language models while reducing cost and improving performance,”

  30. [30]

    Llm-topla: Efficient llm ensemble by maximising diversity,

    S. F. Tekin, F. Ilhan, T. Huang, S. Hu, and L. Liu, “Llm-topla: Efficient llm ensemble by maximising diversity,” 2024. [Online]. Available: https://arxiv.org/abs/2410.03953

    Pith/arXiv arXiv 2024

  31. [31]

    Thriftllm: On cost-effective selection of large language models for classification queries,

    K. Huang, Y . Shi, D. Ding, Y . Li, Y . Fei, L. Lakshmanan, and X. Xiao, “Thriftllm: On cost-effective selection of large language models for classification queries,”Proc. VLDB Endow., vol. 18, no. 11, p. 4410–4423, Jul. 2025. [Online]. Available: https://doi.org/10.14778/3749646.3749702

    arXiv 2025

  32. [32]

    Towards optimizing the costs of llm usage,

    S. Shekhar, T. Dubey, K. Mukherjee, A. Saxena, A. Tyagi, and N. Kotla, “Towards optimizing the costs of llm usage,” 2024. [Online]. Available: https://arxiv.org/abs/2402.01742

    Pith/arXiv arXiv 2024

  33. [33]

    Querying large language models with sql,

    M. Saeed, N. D. Cao, and P. Papotti, “Querying large language models with sql,” inInternational Conference on Extending Database Technology, 2023. [Online]. Available: https://api.semanticscholar.org/ CorpusID:257913347

    2023

  34. [34]

    Abacus: A cost-based optimizer for semantic operator systems,

    M. Russo, S. Sudhir, G. Vitagliano, C. Liu, T. Kraska, S. Madden, and M. J. Cafarella, “Abacus: A cost-based optimizer for semantic operator systems,”ArXiv, vol. abs/2505.14661, 2025. [Online]. Available: https://api.semanticscholar.org/CorpusID:278768629

    arXiv 2025

  35. [35]

    The stretto execution engine for llm-augmented data systems,

    G. Sanmartino, M. Urban, P. Papotti, and C. Binnig, “The stretto execution engine for llm-augmented data systems,” 2026. [Online]. Available: https://arxiv.org/abs/2602.04430

    arXiv 2026

  36. [36]

    Does size matter? on the influence of ensemble size on constructing ensembles of dispatching rules,

    M. Durasevic, F. J. Gil-Gala, and D. Jakobovi ´c, “Does size matter? on the influence of ensemble size on constructing ensembles of dispatching rules,” inProceedings of the Companion Conference on Genetic and Evolutionary Computation, ser. GECCO ’23 Companion. New York, NY , USA: Association for Computing Machinery, 2023, p. 559–562. [Online]. Available: ...

    arXiv 2023

  37. [37]

    ML.ENERGY leaderboard,

    J.-W. Chung, J. Liu, Z. Wu, Y . Xia, and M. Chowdhury, “ML.ENERGY leaderboard,” https://ml.energy/leaderboard, 2023

    2023

  38. [38]

    A fast and elitist multiobjective genetic algorithm: Nsga-ii,

    K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan, “A fast and elitist multiobjective genetic algorithm: Nsga-ii,”IEEE Transactions on Evo- lutionary Computation, vol. 6, no. 2, pp. 182–197, 2002

    2002

  39. [39]

    Opti-q extended version,

    A. Hamid and coauthors, “Opti-q extended version,” https://github.com /Aamir7693/Opti-Q/blob/main/extended version/extended version.pdf, 2026, extended version

    2026

  40. [40]

    Zeus: Understanding and optimizing gpu energy consumption of dnn training,

    J. You, J. Chung, and M. Chowdhury, “Zeus: Understanding and optimizing gpu energy consumption of dnn training,”ArXiv, vol. abs/2208.06102, no. n/a, p. n/a, 2022. [Online]. Available: https://api.semanticscholar.org/CorpusID:251554526

    Pith/arXiv arXiv 2022

  41. [41]

    A fast randomized algorithm for multi- objective query optimization,

    I. Trummer and C. Koch, “A fast randomized algorithm for multi- objective query optimization,” inProceedings of the 2016 International Conference on Management of Data, 2016, pp. 1737–1752

    2016

  42. [42]

    [Online]

    Gemma Team, “Gemma,” Kaggle Model Artifact, 2024. [Online]. Available: https://www.kaggle.com/m/3301

    2024

  43. [43]

    Chatqa: Surpassing gpt-4 on conversational qa and rag,

    Z. Liu, W. Ping, R. Roy, P. Xu, C. Lee, M. Shoeybi, and B. Catanzaro, “Chatqa: Surpassing gpt-4 on conversational qa and rag,”arXiv preprint arXiv:2401.10225, vol. n/a, no. n/a, p. n/a, 2024

    Pith/arXiv arXiv 2024

  44. [44]

    Qwen3 technical report,

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lvet al., “Qwen3 technical report,”arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  45. [45]

    Phi-4 technical report,

    M. Abdin, J. Aneja, H. Behl, S. Bubeck, R. Eldan, S. Gunasekar, M. Harrison, R. J. Hewett, M. Javaheripi, P. Kauffmann, J. R. Lee, Y . T. Lee, Y . Li, W. Liu, C. C. T. Mendes, A. Nguyen, E. Price, G. de Rosa, O. Saarikivi, A. Salim, S. Shah, X. Wang, R. Ward, Y . Wu, D. Yu, C. Zhang, and Y . Zhang, “Phi-4 technical report,” 2024. [Online]. Available: http...

    Pith/arXiv arXiv 2024

  46. [46]

    Mistral 7b,

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed, “Mistral 7b,” 2023. [Online]. Available: https://arxiv.org/abs/2310.06825

    Pith/arXiv arXiv 2023

  47. [47]

    Benchmarking llms via uncertainty quantification,

    F. Ye, M. Yang, J. Pang, L. Wang, D. Wong, E. Yilmaz, S. Shi, and Z. Tu, “Benchmarking llms via uncertainty quantification,”Advances in Neural Information Processing Systems, vol. 37, pp. 15 356–15 385, 2024

    2024

  48. [48]

    Bertopic: Neural topic modeling with a class-based tf-idf procedure,

    M. Grootendorst, “Bertopic: Neural topic modeling with a class-based tf-idf procedure,”arXiv preprint arXiv:2203.05794, vol. n/a, no. n/a, p. n/a, 2022

    Pith/arXiv arXiv 2022

  49. [49]

    Umap: Uniform manifold approximation and projection for dimension reduction,

    L. McInnes, J. Healy, and J. Melville, “Umap: Uniform manifold approximation and projection for dimension reduction,” 2020. [Online]. Available: https://arxiv.org/abs/1802.03426

    Pith/arXiv arXiv 2020

  50. [50]

    hdbscan: Hierarchical density based clustering

    L. McInnes, J. Healy, S. Astelset al., “hdbscan: Hierarchical density based clustering.”J. Open Source Softw., vol. 2, no. 11, p. 205, 2017

    2017

  51. [51]

    Large lan- guage models are zero-shot reasoners,

    T. Kojima, S. S. Gu, M. Reid, Y . Matsuo, and Y . Iwasawa, “Large lan- guage models are zero-shot reasoners,”Advances in neural information processing systems, vol. 35, pp. 22 199–22 213, 2022

    2022

  52. [52]

    Language mod- els are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askellet al., “Language mod- els are few-shot learners,”Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020

    1901

  53. [53]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” inProceedings of the 40th Annual Meeting of the Association for Computational Linguistics, P. Isabelle, E. Charniak, and D. Lin, Eds. Philadelphia, Pennsylvania, USA: Association for Computational Linguistics, Jul. 2002, pp. 311–318. [Online]....

    2002

  54. [54]

    ROUGE: A package for automatic evaluation of summaries,

    C.-Y . Lin, “ROUGE: A package for automatic evaluation of summaries,” inText Summarization Branches Out. Barcelona, Spain: Association for Computational Linguistics, Jul. 2004, pp. 74–81. [Online]. Available: https://aclanthology.org/W04-1013/

    2004

  55. [55]

    Harnessing multiple large language models: A survey on llm ensemble,

    Z. Chen, J. Li, P. Chen, Z. Li, K. Sun, Y . Luo, Q. Mao, D. Yang, H. Sun, and P. S. Yu, “Harnessing multiple large language models: A survey on llm ensemble,” 2025. [Online]. Available: https://arxiv.org/abs/2502.18036

    Pith/arXiv arXiv 2025

  56. [2023]

    Available: https://arxiv.org/abs/2305.05176

    [Online]. Available: https://arxiv.org/abs/2305.05176

    Pith/arXiv arXiv