Pith. sign in

REVIEW 4 major objections 5 minor 76 references

syftr: Pareto-Optimal Generative AI

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read syftr uses multi-objective Bayesian search to find RAG flows that are on average 9 times cheaper than the most accurate flows while preserving most of their accuracy, and that dominate default flows by 6% accuracy at equal cost or 37%…

desk verdict Solid engineering contribution that needs a holdout evaluation and a judge-fidelity check before the headline numbers can be trusted. read the letter →

arxiv 2505.20266 v1 pith:TAAMIKDY submitted 2025-05-26 cs.AI cs.LG

Alexander Conway , Debadeepta Dey , Stefan Hackmann , Matthew Hausknecht , Michael Schmidt , Mark Steadman , Nick Volynets This is my paper · ORCID
classification cs.AIcs.LG
keywords retrieval-augmentedgenerationmulti-objectiveBayesianoptimizationParetofrontieragenticRAGflowsLLM-as-a-judgeearlystoppingpipelinecost-accuracytradeoff
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces syftr, a system that treats RAG pipeline design as a multi-objective search problem: instead of hand-picking splitters, retrievers, LLMs, and agentic strategies, it searches a hierarchical space of over $10^{23}$ possible flows and returns a Pareto frontier trading task accuracy against cost. The claim is that this automated search beats manual defaults: on multiple RAG benchmarks, syftr finds flows that are on average about 9 times cheaper while preserving most of the accuracy of the most accurate flows, and flows that dominate a standard RAG library's default by 6% more accuracy at equal cost or 37% lower cost at equal accuracy. If true, it turns a costly expert design task into a data-driven process and gives practitioners the full tradeoff curve, not just one configuration. The paper also introduces a pruning mechanism that stops evaluating flows that cannot improve the frontier, which is what makes the search affordable.

What carries the argument

The central machinery is multi-objective Tree-of-Parzen Estimators (MO-TPE), a Bayesian optimization method that models the distribution of good versus poor configurations rather than the objective surface, applied to a hierarchical search space of RAG flows. Search is guided by expected hypervolume improvement, and a Pareto-Pruner computes confidence intervals on accuracy and cost for each running evaluation and terminates any trial whose upper confidence point falls below the current frontier. The paper also uses an LLM-as-a-judge scoring protocol, selected after comparing ten judge configurations against human labels, to turn QA answers into a per-flow accuracy signal.

What would settle it

Take one of the reported frontier flows and run it on the held-out partition, scoring answers by human labelers rather than the LLM judge; if the frontier flows do not outperform the default flow by the claimed margins, or if judge scores and human labels disagree sharply (as the reported kappa of 0.29 suggests they might), the central claim fails. A cheaper check: compare judge scores to human labels on a fresh sample of flow outputs; if the judge's agreement with humans is as low as kappa 0.29, the optimizer may be optimizing judge preferences, not answer quality.

Watch

Extended reading notes

Core claim

The central claim is that multi-objective Bayesian optimization can efficiently discover Pareto-optimal RAG flows in a search space containing over $10^{23}$ configurations. Using MO-TPE to guide sampling and an early-stopping Pareto-Pruner to cut off unpromising evaluations, syftr constructs and evaluates flows on question-answer pairs, scoring answers with an LLM judge, and reports frontiers across six benchmarks. On average the optimizer finds flows roughly nine times cheaper than the most accurate flows at comparable accuracy, and Pareto-dominant relative to a default non-agentic RAG flow (about 6% higher accuracy at the same cost, 37% cheaper at the same accuracy). The paper also reports that non-agentic flows dominate the frontiers, that smaller models appear frequently, and that marginal accuracy gains flatten sharply as cost rises.

Load-bearing premise

The load-bearing premise is that the LLM-as-a-judge score used as the optimization objective is a faithful proxy for true task accuracy; if the judge is biased or the test partition is overfit, the reported Pareto frontiers and the 6% and 37% advantages will not reproduce in deployment.

Editorial extensions

If this is right

  • Practitioners can query the resulting Pareto frontier to pick a flow that meets a latency or budget constraint, rather than guessing a configuration.
  • The optimized flows generalize poorly across datasets, so per-dataset search matters more than a single universal RAG recipe.
  • The large-model study implies that upgrading LLM sizes in an optimized flow can yield large accuracy gains but enormous cost multipliers, and not every small-model frontier flow remains on the large-model frontier.
  • The transfer-seeding result implies that prior searches can warm-start new datasets, reducing cumulative cost.
  • The framework's design allows new modules to be added and evaluated holistically, making it easier to assess a new LLM or retriever across diverse flows.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that the same machinery could optimize other objectives, such as latency or a composite score, which the latency study begins to demonstrate.
  • A natural extension the authors do not explore is optimizing the judge itself inside the loop: since the judge is the objective, its biases will be amplified by the optimizer, so selecting judge configurations jointly with flows may be necessary for real-world robustness.
  • An outside reader should test whether the reported 6% and 37% advantages persist on holdout partitions with human-labeled accuracy, because the judge's agreement with human labels is low (Cohen's kappa of 0.29) even though Pearson correlation is high.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces syftr, a system that searches a hierarchical space of roughly 10^23 retrieval-augmented generation (RAG) flow configurations using multi-objective Bayesian optimization (MO-TPE) to approximate the Pareto frontier between task accuracy and monetary cost. The search space includes non-agentic RAG and several agentic flows, with modules for retrievers, splitters, embedding models, rerankers, HyDE, prompt templates, and synthesizing LLMs. The paper proposes a Pareto-Pruner for early stopping, static/random/transfer seeding strategies, and a latency-optimization variant. Across six RAG benchmark datasets, the authors report that syftr finds flows that are on average 6% more accurate at the same cost as a LlamaIndex-style default flow, or 37% cheaper at the same accuracy, and that the abstract headline claims an approximately 9x cost reduction relative to the most accurate Pareto-frontier flows. Additional studies examine upgrading to larger LLMs, agentic-only search, transfer seeding, and a latency/accuracy tradeoff. The evaluation uses an LLM-as-a-judge accuracy metric that was calibrated against 447 author-labeled responses.

Significance. If the reported results hold, syftr would be a practically valuable tool: it automates a costly manual design process over a very large compositional flow space, jointly optimizing accuracy and cost, and its appendices provide substantial engineering detail (seeding, pruning, transfer learning, infrastructure, and evaluation studies). The use of established MO-TPE machinery, the large search space, and the inclusion of agentic and non-agentic flows are genuine strengths, as is the explicit discussion of evaluation-judge sensitivity. However, the central quantitative claims currently rest on an evaluation protocol that optimizes and reports on the same test partition, with no holdout evaluation, no repeated runs, and an LLM judge with modest inter-annotator agreement. The contribution is therefore best viewed as a promising system demonstration whose headline numbers are not yet established by the evidence in the manuscript.

major comments (4)
  1. [§5 and Appendix A6] The main quantitative claims (6% accuracy gain, 37% cost reduction, and the Pareto frontiers in Figs. 5, A4, A9) are obtained by selecting flows on the same test partition used for optimization. Appendix A6 states: "Flow evaluation during optimization always uses the test partition" and "We report accuracy numbers for flows evaluated on the test set, and set aside the holdout partition for future use." This is a selection-on-the-evaluation-set protocol: the reported deltas are order statistics from roughly 500 trials over a 10^23 space, and they are expected to be optimistically biased. To support the claims, the authors should evaluate the final Pareto-optimal flows on the reserved holdout partitions and report those numbers, ideally with multiple optimization seeds or at least bootstrapped confidence intervals.
  2. [Abstract and §6] The abstract's headline claim that syftr finds flows "on average approximately 9 times cheaper" is not reconciled with the body's claim of "37% cheaper for the same baseline accuracy." The two statements refer to different baselines (the most accurate Pareto-frontier flow versus a fixed default flow), but this is never made explicit, and the factor-of-9 figure does not appear in the results section or in Fig. A9. The authors should either define the comparison precisely, report the supporting per-dataset numbers, or remove the 9x claim from the abstract.
  3. [§5, Appendix A7, Table A8] The optimization objective is an LLM-as-a-judge score, and the selected Default-Prompt Random LLM configuration has Pearson correlation 0.84 but Cohen's kappa 0.29 against human labels on 447 responses. A kappa of 0.29 indicates only fair agreement beyond chance on the pass/fail decision, and no independent holdout evaluation of the judge is reported. Because the optimizer maximizes this judge score, the Pareto frontier may be a frontier in judge score rather than in human-perceived accuracy. The paper should report the judge's per-dataset agreement, the distribution of kappa across judge LLMs, and a human evaluation (or at least a holdout judge evaluation) of the final selected flows.
  4. [§6 and Appendix A12] The multi-dataset and large-model studies report a single run per dataset with no error bars or repeated-seed analysis. Given that LLM outputs and cost measurements are stochastic, the reported Pareto frontier points and the 17.3 percentage-point average accuracy improvement in Table A9 carry unknown variance. The authors should add repeated runs (or at least variance estimates from the sampled trials) to make the comparative claims statistically grounded.
minor comments (5)
  1. [§1] The text contains typos such as "introducesyftr" and "Parto-frontier" in the caption of Fig. 4, and "clause-3.5-sonnet" in Section 1; these should be corrected.
  2. [Abstract] The abstract ends with "/githubCode", which appears to be a placeholder rather than an actual repository link; if code is available, a full URL should be provided.
  3. [§6] The phrase "syftr finds Pareto-dominant flows" is imprecise: a single flow cannot be Pareto-dominant relative to a baseline unless it dominates on both objectives; the intended meaning (a flow that dominates the baseline point) should be stated explicitly.
  4. [Appendix A2] The Pareto-Pruner uses log-normal and normal distribution parameters fitted per dataset, but the fitting procedure and the data used for fitting are not described; a sentence on the fitting set and a sensitivity analysis over the z-score and distribution choices would clarify the robustness of the pruning method.
  5. [References] Some references are incomplete or informal (e.g., HuggingFace inference endpoints documentation is cited without author or year), which may hinder reproducibility.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation; central claims are empirical search results, though the test-partition protocol and self-referential judge calibration create mild self-reference and validity risk.

full rationale

The paper's derivation chain is empirical rather than formal: Bayesian optimization over a hierarchical RAG flow space, with Pareto-Pruner early stopping and LLM-as-a-judge scoring. The reported Pareto frontiers and the 6%/37% baseline comparisons are observed statistics of evaluated flows, not quantities equated to optimizer inputs by construction. The main self-referential element is the LLM judge: Appendix A7 calibrates ten judge configurations against 447 responses labeled by the authors, and the chosen Random LLM evaluator has Pearson 0.84 but Cohen's kappa 0.29 (Table A8). This is a validity concern for the accuracy objective, but the judge score is not defined as the final reported accuracy; it is an external (though author-labeled) calibration signal. The Pareto-Pruner in Section 4 fits log-normal cost and normal accuracy distributions to observed trial data (Appendix A2), but those distributions only drive early stopping and are not renamed as headline predictions. The clearest admitted limitation is in Appendix A6: 'Flow evaluation during optimization always uses the test partition... We report accuracy numbers for flows evaluated on the test set, and set aside the holdout partition for future use.' That is a selection-on-the-evaluation-set protocol, which can inflate claimed gains and should be weighed as a robustness risk, but it is not a circular step because the reported accuracies are actual evaluation statistics rather than fitted parameters or derived quantities identical to inputs. Self-citations in related work ([30], [31], [35]) are background references and are not load-bearing for syftr's central claims. No uniqueness theorem, ansatz-smuggling citation, or renaming of a known result is used to force the main conclusions. Accordingly, the paper is not significantly circular; the concerns are methodological validity issues rather than definitional circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the reliability of the LLM-judge objective and on the representativeness of the test partition; neither is independently verified. The pruner adds fitted distribution parameters used for early stopping, but these do not directly generate the claimed tradeoffs.

free parameters (3)
  • Cost distribution parameters (log-normal mu_c, sigma_c) per dataset = Several per-dataset fits, e.g., FinanceBench mu=5.45, sigma=1.76
    Fitted to observed trial costs in Appendix A2 and used by the Pareto-Pruner to compute confidence intervals for early stopping.
  • Accuracy distribution parameters (normal mu_a, sigma_a) per dataset = Several per-dataset fits, e.g., FinanceBench mu=0.29, sigma=0.12
    Fitted to observed trial accuracies in Appendix A2 and used by the Pareto-Pruner.
  • z-score for pruning confidence = 1.645 (90% confidence)
    Chosen by the authors to control Pareto-Pruner sensitivity; not fitted to data.
assumptions (5)
  • domain assumption LLM-as-a-judge with the Random LLM configuration approximates human judgment well enough to serve as the optimization objective.
    Validated on 447 responses labeled by the authors themselves; Pearson correlation 0.84 but Cohen's kappa only 0.29 (Appendix A7, Table A8).
  • domain assumption The log-normal and normal models for cost and accuracy are adequate for the pruner's confidence intervals.
    Appendix A2 shows approximations but no goodness-of-fit tests; pruning decisions depend on these distributions.
  • domain assumption MO-TPE with Optuna's EHVI acquires useful candidates in this high-dimensional, stochastic, hierarchical space.
    The paper relies on prior work [47,48] and does not compare MO-TPE to random or grid search over the same space.
  • domain assumption Flows selected and evaluated on the test partition will generalize to deployment.
    Section 5 states optimization is done on the test partition and the holdout is reserved, but no holdout results are reported.
  • domain assumption Benchmark ground-truth answers are correct and the evaluation splits are representative.
    Standard public benchmarks are used, but CRAG is modified substantially and DRDocs is proprietary.

how reviews work

0 comments
Cite this review

Pith. "Pith review of syftr: Pareto-Optimal Generative AI." pith.science (2026). https://pith.science/paper/TAAMIKDY

@misc{pith2026250520266,
  author       = {Pith},
  title        = {Pith review of: syftr: Pareto-Optimal Generative AI},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TAAMIKDY}},
  note         = {Machine review of arXiv:2505.20266}
}
read the original abstract

Retrieval-Augmented Generation (RAG) pipelines are central to applying large language models (LLMs) to proprietary or dynamic data. However, building effective RAG flows is complex, requiring careful selection among vector databases, embedding models, text splitters, retrievers, and synthesizing LLMs. The challenge deepens with the rise of agentic paradigms. Modules like verifiers, rewriters, and rerankers-each with intricate hyperparameter dependencies have to be carefully tuned. Balancing tradeoffs between latency, accuracy, and cost becomes increasingly difficult in performance-sensitive applications. We introduce syftr, a framework that performs efficient multi-objective search over a broad space of agentic and non-agentic RAG configurations. Using Bayesian Optimization, syftr discovers Pareto-optimal flows that jointly optimize task accuracy and cost. A novel early-stopping mechanism further improves efficiency by pruning clearly suboptimal candidates. Across multiple RAG benchmarks, syftr finds flows which are on average approximately 9 times cheaper while preserving most of the accuracy of the most accurate flows on the Pareto-frontier. Furthermore, syftr's ability to design and optimize allows integrating new modules, making it even easier and faster to realize high-performing generative AI pipelines.

Figures

Figures reproduced from arXiv: 2505.20266 by the authors.

Figure 1
Figure 1. Given a grounding corpus, syftr searches over more than 1023 unique RAG flows to find a Pareto-frontier (optimal tradeoff curve) between task accuracy and cost. Generation (RAG) addresses these challenges by dynamically integrating external knowledge into model outputs, improving accuracy and reliability by grounding responses in verifiable sources [3, 4]. To operationalize RAG, generative AI flows (or pipelines) or… view at source ↗
Figure 2
Figure 2. In this simplified view of the canonical RAG flow we term as “vanilla RAG”, the developer [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. syftr RAG hierarchical search space includes 5 top-level flows – 4 agentic and 1 non-agentic with a total of 1023 unique flows. The agentic flows use the RAG flow as a subroutine while adding their own unique hyperparameters. 3 Search Space The syftr search space ( [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Pareto-Pruner estimates confidence intervals around task accuracy and cost for a given flow, and will early-terminate flows whose upper-left confidence point (p) falls below the current Pareto-frontier. c is the P80 cost, a is the average accuracy, L is the number of s…
Figure 5
Figure 5. Figure 5: Multi-Dataset Study: Pareto-frontier for InfiniteBench; See Appendix [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

76 extracted references · 54 canonical work pages

  1. [1]

    Retrieval-augmented generation for knowledge-intensive nlp tasks, 2021

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K¨uttler, Mike Lewis, Wen tau Yih, Tim Rockt¨aschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks, 2021

    work page 2021

  2. [2]

    Retrieval augmenta- tion reduces hallucination in conversation

    Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. Retrieval augmenta- tion reduces hallucination in conversation. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3784–3803, Punta Cana, Dominican Republic, November 2021. A...

    work page 2021

  3. [3]

    Realm: retrieval- syftr 11 augmented language model pre-training

    Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: retrieval- syftr 11 augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, ICML’20. JMLR.org, 2020

    work page 2020

  4. [4]

    Improving language models by retrieving from trillions of tokens

    Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, ...

    work page 2022

  5. [5]

    LangChain, October 2022

    Harrison Chase. LangChain, October 2022

    2022

  6. [6]

    Autogen: Enabling next-gen llm applications via multi-agent conversation, 2023

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. Autogen: Enabling next-gen llm applications via multi-agent conversation, 2023

    2023

  7. [7]

    Haystack: the end-to-end NLP framework for pragmatic builders, November 2019

    Malte Pietsch, Timo M¨oller, Bogdan Kostic, Julian Risch, Massimiliano Pippi, Mayank Jobanputra, Sara Zanzottera, Silvano Cerza, Vladimir Blagojevic, Thomas Stadelmann, Tanay Soni, and Sebastian Lee. Haystack: the end-to-end NLP framework for pragmatic builders, November 2019

    work page 2019

  8. [8]

    crewAI: Collaborative Research and Workflows AI, 2023

    crewAI Inc. crewAI: Collaborative Research and Workflows AI, 2023

    work page 2023

Show all 76 references
  1. [9]

    LlamaIndex, November 2022

    Jerry Liu. LlamaIndex, November 2022

    2022

  2. [10]

    Approximate nearest neighbor search in high dimensions, 2018

    Alexandr Andoni, Piotr Indyk, and Ilya Razenshteyn. Approximate nearest neighbor search in high dimensions, 2018

    2018

  3. [11]

    Mteb: Massive text embedding benchmark, 2023

    Niklas Muennighoff, Nouamane Tazi, Lo ¨ıc Magne, and Nils Reimers. Mteb: Massive text embedding benchmark, 2023

    2023

  4. [12]

    Chonkie: A fast feature-full chunking library for rag bots

    Bhavnick Minhas. Chonkie: A fast feature-full chunking library for rag bots. https://github. com/bhavnick/chonkie, 2024

    2024

  5. [13]

    The probabilistic relevance framework: Bm25 and beyond

    Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389, 2009

    2009

  6. [14]

    Rag-fusion: A new take on retrieval augmented generation

    Zackary Rackauckas. Rag-fusion: A new take on retrieval augmented generation. International Journal on Natural Language Computing, 13(1):37–47, February 2024. syftr 12

    2024

  7. [15]

    Pinecone: A vector database for machine learning applications, 2025

    Pinecone Systems Inc. Pinecone: A vector database for machine learning applications, 2025. Accessed: 2025-01-04

    2025

  8. [16]

    The faiss library, 2024

    Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre- Emmanuel Mazar´e, Maria Lomeli, Lucas Hosseini, and Herv´e J´egou. The faiss library, 2024

    2024

  9. [17]

    Neo4j: The World’s Leading Graph Database

    Neo4j, Inc. Neo4j: The World’s Leading Graph Database . Neo4j, Inc., n.d. Available at https: //neo4j.com

  10. [18]

    Agentic retrieval- augmented generation for time series analysis, 2024

    Chidaksh Ravuru, Sagar Srinivas Sakhinana, and Venkataramana Runkana. Agentic retrieval- augmented generation for time series analysis, 2024

    2024

  11. [19]

    Building a rag pipeline is difficult, 2024

    Nikhil Bysani and Ofer Mendelevitch. Building a rag pipeline is difficult, 2024. Accessed: 2025-01-04

    2024

  12. [20]

    Large language models are better reasoners with self-verification, 2023

    Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. Large language models are better reasoners with self-verification, 2023

    2023

  13. [21]

    Precise zero-shot dense retrieval without relevance labels, 2022

    Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. Precise zero-shot dense retrieval without relevance labels, 2022

    2022

  14. [22]

    Re2g: Retrieve, rerank, generate, 2022

    Michael Glass, Gaetano Rossiello, Md Faisal Mahbub Chowdhury, Ankita Rajaram Naik, Peng- shan Cai, and Alfio Gliozzo. Re2g: Retrieve, rerank, generate, 2022

    2022

  15. [23]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters, 2024

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters, 2024

    2024

  16. [24]

    Convex Optimization

    Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, 2004

    2004

  17. [25]

    Bayesian Optimization

    Roman Garnett. Bayesian Optimization. Cambridge University Press, 2023

    2023

  18. [26]

    Automated Machine Learning - Methods, Systems, Challenges

    Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren, editors. Automated Machine Learning - Methods, Systems, Challenges. Springer, 2019

    2019

  19. [27]

    Auto-sklearn 2.0: Hands-free automl via meta-learning, 2022

    Matthias Feurer, Katharina Eggensperger, Stefan Falkner, Marius Lindauer, and Frank Hutter. Auto-sklearn 2.0: Hands-free automl via meta-learning, 2022

    2022

  20. [28]

    Datarobot: Automated machine learning, 2012

    DataRobot. Datarobot: Automated machine learning, 2012

    2012

  21. [29]

    On hyperparameter optimization of machine learning algorithms: Theory and practice

    Li Yang and Abdallah Shami. On hyperparameter optimization of machine learning algorithms: Theory and practice. Neurocomputing, 415:295–316, November 2020

    2020

  22. [30]

    Neural architecture search: Insights from 1000 papers, 2023

    Colin White, Mahmoud Safari, Rhea Sukthanker, Binxin Ru, Thomas Elsken, Arber Zela, De- badeepta Dey, and Frank Hutter. Neural architecture search: Insights from 1000 papers, 2023. syftr 13

    2023

  23. [31]

    Litetransform- ersearch: Training-free neural architecture search for efficient language models

    Mojan Javaheripi, Gustavo de Rosa, Subhabrata Mukherjee, Shital Shah, Tomasz Religa, Caio Ce- sar Teodoro Mendes, Sebastien Bubeck, Farinaz Koushanfar, and Debadeepta Dey. Litetransform- ersearch: Training-free neural architecture search for efficient language models. In S. Ko...

    2022

  24. [32]

    Advancing the accuracy-efficiency frontier with llama 3.1 and nemotron 51b, 2024

    Akhiad Bercovich and Udi Karpas. Advancing the accuracy-efficiency frontier with llama 3.1 and nemotron 51b, 2024

    2024

  25. [33]

    Autorag: Automated framework for optimization of retrieval augmented generation pipeline, 2024

    Dongkyu Kim, Byoungwook Kim, Donggeon Han, and Matou ˇs Eibich. Autorag: Automated framework for optimization of retrieval augmented generation pipeline, 2024

    2024

  26. [34]

    Learning In Modular Systems

    David Bradley. Learning In Modular Systems. PhD thesis, Carnegie Mellon University, Pittsburgh, PA, May 2010

    2010

  27. [35]

    Metareasoning in modular software systems: On-the-fly configuration using reinforcement learning with rich contextual representations

    Aditya Modi, Debadeepta Dey, Alekh Agarwal, Adith Swaminathan, Besmira Nushi, Sean Andrist, and Eric Horvitz. Metareasoning in modular software systems: On-the-fly configuration using reinforcement learning with rich contextual representations. In Proceedings of the AAAI Confe...

    2020

  28. [36]

    Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. Dspy: Compiling declarative language model calls into self-impro...

    2023

  29. [37]

    Trace is the next autodiff: Generative optimization with rich feedback, execution traces, and llms, 2024

    Ching-An Cheng, Allen Nie, and Adith Swaminathan. Trace is the next autodiff: Generative optimization with rich feedback, execution traces, and llms, 2024

    2024

  30. [38]

    Textgrad: Automatic” differentiation” via text

    Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. Textgrad: Automatic” differentiation” via text. arXiv preprint arXiv:2406.07496, 2024

    2024 arXiv

  31. [39]

    Le, Denny Zhou, and Xinyun Chen

    Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V . Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers, 2024

    2024

  32. [40]

    Parameswaran, and Eugene Wu

    Shreya Shankar, Tristan Chambers, Tarak Shah, Aditya G. Parameswaran, and Eugene Wu. Docetl: Agentic query rewriting and evaluation for complex document processing, 2024

    2024

  33. [41]

    The cascades framework for query optimization

    Goetz Graefe. The cascades framework for query optimization. IEEE Data Eng. Bull., 18(3):19–29, 1995

    1995

  34. [42]

    Kangda Wei, Dawn Lawrie, Benjamin Van Durme, Yunmo Chen, and Orion Weller. When do decompositions help for machine reading? In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 359...

    2023

  35. [43]

    Critic: Large language models can self-correct with tool-interactive critiquing, 2024

    Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. Critic: Large language models can self-correct with tool-interactive critiquing, 2024

    2024

  36. [44]

    React: Synergizing reasoning and acting in language models, 2023

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2023

    2023

  37. [45]

    Lan- guage agent tree search unifies reasoning, acting, and planning in language models

    Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. Lan- guage agent tree search unifies reasoning, acting, and planning in language models. InProceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024

    2024

  38. [46]

    Accessed: 2025-01-15

    Hugging face inference endpoints documentation. Accessed: 2025-01-15

    2025

  39. [47]

    Multiobjective tree- structured parzen estimator for computationally expensive optimization problems

    Yoshihiko Ozaki, Yuki Tanigaki, Shuhei Watanabe, and Masaki Onishi. Multiobjective tree- structured parzen estimator for computationally expensive optimization problems. In Proceedings of the 2020 genetic and evolutionary computation conference, pages 533–541, 2020

    2020

  40. [48]

    Multi- objective tree-structured parzen estimator

    Yoshihiko Ozaki, Yuki Tanigaki, Shuhei Watanabe, Masahiro Nomura, and Masaki Onishi. Multi- objective tree-structured parzen estimator. Journal of Artificial Intelligence Research, 73:1209–1250, 2022

    2022

  41. [49]

    V . Pareto. Cours D’Economie Politique. F. Rouge, Lausanne, Switzerland, 1896

  42. [50]

    Algorithms for hyper-parameter optimization

    James Bergstra, R´emi Bardenet, Yoshua Bengio, and Bal´azs K´egl. Algorithms for hyper-parameter optimization. In J. Shawe-Taylor, R. Zemel, P . Bartlett, F. Pereira, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 24. Curran Associates,...

    2011

  43. [51]

    Optuna: A next-generation hyperparameter optimization framework, 2019

    Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework, 2019

    2019

  44. [52]

    Hotpotqa: A dataset for diverse, explainable multi-hop question answering

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600, 2018

    2018 arXiv

  45. [53]

    Financebench: A new benchmark for financial question answering

    Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. Financebench: A new benchmark for financial question answering. arXiv preprint arXiv:2311.11944, 2023

    2023 arXiv

  46. [54]

    Crag – comprehensive rag benchmark

    Xiao Yang, Kai Sun, Hao Xin, Yushi Sun, Nikita Bhalla, Xiangsen Chen, Sajal Choudhary, Rongze Daniel Gui, Ziran Will Jiang, Ziyu Jiang, Lingkun Kong, Brian Moran, Jiaqi Wang, Yifan Ethan Xu, An Yan, Chenyu Yang, Eting Yuan, Hanwen Zha, Nan Tang, Lei Chen, Nicolas Scheffer, Yue...

    2024 arXiv

  47. [55]

    ∞bench: Extending long context evaluation beyond 100k tokens, 2024

    Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. ∞bench: Extending long context evaluation beyond 100k tokens, 2024

    2024

  48. [56]

    Aryn — document processing for etl, rag, and analytics powered by ai

    Aryn. Aryn — document processing for etl, rag, and analytics powered by ai

  49. [57]

    html2text: Convert html into markdown-formatted text, 2004

    Aaron Swartz. html2text: Convert html into markdown-formatted text, 2004. Version 2024.2.26

    2004

  50. [58]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623, 2023

    2023

  51. [59]

    Llm evaluators recognize and favor their own generations

    Arjun Panickssery, Samuel R Bowman, and Shi Feng. Llm evaluators recognize and favor their own generations. arXiv preprint arXiv:2404.13076, 2024

    2024 arXiv

  52. [60]

    Justice or prejudice? quantifying biases in llm-as-a-judge

    Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, et al. Justice or prejudice? quantifying biases in llm-as-a-judge. arXiv preprint arXiv:2410.02736, 2024

    arXiv

  53. [61]

    Amazon q

    Amazon Web Services. Amazon q. https://aws.amazon.com/q/, 2023. Accessed: 2025-03- 31

    2023

  54. [62]

    Azure OpenAI Assistants (Preview), 2025

    Microsoft. Azure OpenAI Assistants (Preview), 2025. Accessed: 2025-03-31

    2025

  55. [63]

    Magentic-one: A generalist multi-agent system for solving complex tasks

    Adam Fourney, Gagan Bansal, Hussein Mozannar, Cheng Tan, Eduardo Salinas, Erkang (Eric) Zhu, Friederike Niedtner, Grace Proebsting, Griffin Bassman, Jack Gerrits, Jacob Alber, Peter Chang, Ricky Loynd, Robert West, Victor Dibia, Ahmed Awadallah, Ece Kamar, Rafah Hosn, and Sale...

    2024

  56. [64]

    Adaptive in-conversation team building for language model agents, 2024

    Linxin Song, Jiale Liu, Jieyu Zhang, Shaokun Zhang, Ao Luo, Shijian Wang, Qingyun Wu, and Chi Wang. Adaptive in-conversation team building for language model agents, 2024

    2024

  57. [65]

    Lightweight reranking for language model generations, 2024

    Siddhartha Jain, Xiaofei Ma, Anoop Deoras, and Bing Xiang. Lightweight reranking for language model generations, 2024

    2024

  58. [66]

    Zero-shot listwise document reranking with a large language model, 2023

    Xueguang Ma, Xinyu Zhang, Ronak Pradeep, and Jimmy Lin. Zero-shot listwise document reranking with a large language model, 2023

    2023

  59. [67]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24827–24837, 2022. syftr 16

    2022

  60. [68]

    Rethinking the role of demonstrations: What makes in-context learning work?, 2022

    Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work?, 2022

    2022

  61. [69]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  62. [70]

    Jordan, and Ion Stoica

    Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I. Jordan, and Ion Stoica. Ray: A distributed framework for emerging ai applications, 2018

    2018

  63. [71]

    Gonzalez, and Ion Stoica

    Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E. Gonzalez, and Ion Stoica. Tune: A research platform for distributed model selection and training, 2018

    2018

  64. [72]

    https://github.com/onnx/onnx/ blob/main/docs/IR.md, 2024

    Open neural network exchange (onnx) specification. https://github.com/onnx/onnx/ blob/main/docs/IR.md, 2024

    2024

  65. [73]

    Bilevel programming for hyperparameter optimization and meta-learning

    Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and meta-learning. In International conference on machine learning, pages 1568–1577. PMLR, 2018

    2018

  66. [74]

    Model-agnostic meta-learning for fast adaptation of deep networks

    Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pages 1126–1135. PMLR, 2017

    2017

  67. [75]

    Meta comprehensive rag benchmark - kdd cup 2024

    AIcrowd. Meta comprehensive rag benchmark - kdd cup 2024. https://www.aicrowd.com/ challenges/meta-comprehensive-rag-benchmark-kdd-cup-2024 , 2024. Accessed: 2025-01-15

    2024

  68. [76]

    distractor

    LlamaIndex. Correctness evaluation example. https://docs.llamaindex.ai/en/stable/ examples/evaluation/correctness_eval/. Accessed: 2025-01-16. syftr 17 A1 Search Space Details Table A1: syftr choices for each module and their search spaces for the RAG workflow. For discrete se...

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.