Pith. sign in

REVIEW 3 major objections 5 minor 63 references

Choosing a different LLM for each step of an agentic workflow—not one model for the whole plan—raises accuracy under the same budget.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 23:55 UTC pith:NKLGZBY6

load-bearing objection Solid systems paper: trie + cascade MNAR correction makes per-invocation agent control practical, with real Pareto gains and huge profiling savings; monotonicity is assumed, not measured. the 3 major comments →

arxiv 2605.23914 v1 pith:NKLGZBY6 submitted 2026-04-09 cs.DC cs.AIcs.MA

VineLM: Trie-Based Fine-Grained Control for Agentic Workflows

Nikos Pagonas , Matthew Lou , Tianyi Peng , Dan Rubenstein , Kostis Kaffes This is my paper
classification cs.DC cs.AIcs.MA
keywords agentic workflowsLLM servingexecution triecascade profilingper-invocation routingNL2SQLcost-latency-accuracyonline replanning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Agentic workflows chain LLM stages with tools and often loop for repair or self-reflection. Today’s managers pick one model per stage template offline and stick with that assignment for the whole request, even when a later repair would be better served by a different model. VineLM instead treats every stage invocation as a separate decision. It organizes all feasible sequences of model choices as a trie of prefixes, sparsely profiles that trie with checkpointing and cascade sampling, and at runtime re-roots the trie after each step so the next model can be chosen from the remaining subtrie given the budget left. On NL2SQL and math-reasoning workflows this finer grain improves the accuracy–cost–latency frontier, delivering up to 18% higher accuracy at the same budget while cutting offline profiling cost by roughly 98–99.8% relative to exhaustive enumeration.

Core claim

Fine-grained, per-invocation model selection over an annotated execution trie yields a strictly better cost–latency–accuracy frontier than static workflow-level plans that bind one model to each stage template and reuse it across loop iterations. Sparse cascade profiling recovers the column means needed for path selection at 1% mean absolute error with only 2% of the cost of exhaustive profiling, and online re-rooting after each stage reduces latency-SLO violations by up to 85%.

What carries the argument

An annotated execution trie whose nodes are prefixes of model choices (including successive loop iterations of the same logical stage). Offline, checkpointing plus cascade decomposition estimates expected accuracy, cost, and latency at each node without full matrix observation; online, the controller re-roots at the realized prefix and searches only the remaining subtrie under the residual budget.

Load-bearing premise

Expected accuracy never falls when extra refinement stages are added, and the offline request sample is large and stationary enough that average path metrics transfer to live traffic.

What would settle it

On a held-out stream of NL2SQL or MathQA requests whose difficulty distribution differs from the profiling set, measure whether VineLM still beats the best static workflow-level plan under the same cost or latency budget; if the accuracy gap disappears or reverses, the transfer claim fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. VineLM is a workflow manager for agentic LLM pipelines that interleave configurable model stages with tools and refinement loops. Unlike coarse managers (e.g., Murakkab) that bind one model per stage template and fix that assignment for the request lifetime, VineLM chooses a model per stage invocation. Feasible executions are represented as an annotated trie of model-choice prefixes; offline, checkpointing plus cascade sampling with an MNAR-aware cascade decomposition (and optional rank-1 smoothing) estimate path accuracy, cost, and latency without exhaustive profiling; online, the controller re-roots after each invocation and replans the remaining subtrie under request-level objectives. On NL2SQL and MathQA workloads the system reports up to ~18% higher accuracy at the same budget, 98–99.8% lower offline profiling cost versus exhaustive profiling, and large reductions in latency-SLO violations via dynamic re-rooting.

Significance. The paper addresses a real and timely gap in agentic serving: refinement loops make per-invocation model choice both combinatorially hard and operationally valuable. The execution-trie formulation, the cascade decomposition that turns MNAR sampling into an exact recursive estimator, and the practical sparse-profiling pipeline (checkpointing + fill-in + low-rank smoothing) are concrete technical contributions. End-to-end gains versus a strong workflow-level baseline, together with a careful estimator ablation (§5.3) and load-aware latency results (§5.4), make the work significant for systems venues if the load-bearing semantic assumptions hold. The profiling-cost reductions (Table 2) are especially useful for practitioners.

major comments (3)
  1. [§3.3, §4.2, Appendix A] §3.3 and the Remark on Monotonicity, together with cascade fill-in and the recursive estimator in §4.2 / Appendix A (Eqs. 1, 4–9), treat path accuracy as prefix-closed and monotone: success at any prefix implies success for every extension, and additional stages never decrease expected accuracy. This underpins subtree fill-in, DFS pruning, and column-mean ranking. For NL2SQL repair and especially MathQA self-reflection, a later stage can overwrite a previously correct answer (over-correction, noisy critique). The paper never reports the empirical rate of correct→incorrect flips, nor does it verify that final-answer accuracy (as opposed to “ever correct under early stop”) is prefix-closed on the evaluation workloads. Without that measurement, both the sparse estimator’s bias claims and the ranking that produces the reported accuracy deltas remain incompletely validated. Please add a direc
  2. [§5.2, Figure 7] The headline accuracy gains under cost SLOs (§5.2, Figure 7) are attributed to “fine-grained control,” but the evaluation does not cleanly separate (i) the enlarged static path space obtained by unrolling loops into distinct per-iteration decisions from (ii) online re-rooting that adapts to realized prefixes and remaining budget. For pure cost/accuracy objectives, re-rooting mainly updates remaining latency; cost annotations are offline expectations. A static fine-grained baseline (choose a full root-to-leaf path at admission from the same annotated trie, no re-rooting) is needed to show how much of the up-to-18% delta is due to mixed-model trajectories versus dynamic adaptation. Without it, the central claim over-attributes gains to the online controller relative to the trie formulation alone.
  3. [§4.2, §5.1–5.2] Offline cascade sampling continues on ground-truth failure (§4.2, Appendix A), while online control flow for NL2SQL is driven by tool feedback (execution errors / empty results) and for MathQA by self-critique or a fixed horizon—neither of which observes ground truth. The paper does not quantify how often GT-failure and online stop signals disagree, nor how that mismatch affects transfer of ¯A from Q to live traffic. Given that the cascade decomposition is derived under GT-conditioned continuation, a short disagreement analysis (or an ablation that profiles with online-like stop signals) is needed to support that the annotated trie ranks suffixes correctly at serving time.
minor comments (5)
  1. [§5.2, Figure 7] Figure 7 is described as accuracy delta over Murakkab but the manuscript text does not state the absolute Murakkab accuracies or the exact cost-SLO grid; adding absolute curves (or a table of selected operating points) would make the 18% claim easier to audit.
  2. [§3.3, Appendix A] Notation for path metrics switches between A(q,p)/¯A(p) and μ(p)/ˆμ in the main text versus Appendix A; a single consistent notation would help.
  3. [§5.4, Table 3] Table 3 reports controller overhead in microseconds/milliseconds; stating the number of trie nodes searched (or worst-case branching) would clarify scaling with depth and |L|.
  4. [§6] Related work on LLM routing/cascades is adequate; a brief note on how VineLM could incorporate prompt-level difficulty signals (mentioned only in passing) would strengthen the discussion of hybrid designs.
  5. [§2–4] Minor typos: “esimates” (§3.4), “an be constrained” (§4.2), “Gemma27B” spacing inconsistencies in §2.2.

Circularity Check

0 steps flagged

No circularity: VineLM's frontier gains and sparse-profiling claims are empirical measurements, not results forced by definition or self-citation.

full rationale

The paper's load-bearing claims are systems/empirical: (i) higher accuracy under the same cost budget vs Murakkab-style workflow-level plans, (ii) column-mean MAE and policy fidelity under sparse cascade profiling vs full profiling, and (iii) latency-SLO violation rates under dynamic re-rooting. Path metrics Ā, C̄, T̄ are estimated from offline request–path executions (with cascade decomposition correcting MNAR observation of conditional successes); the cascade identity μ(ℓ1,ℓ2)=μ(ℓ1)+(1−μ(ℓ1))·Pr[ℓ2|ℓ1 fails] is a consequence of the paper's own success semantics, not a fit of the reported end-to-end accuracy delta. Online control re-searches the annotated trie using realized latency; evaluation measures actual accuracy, cost, and SLO violations on the workloads. There is no uniqueness theorem, ansatz, or prior result by the same authors that forces the 18% gain or the 98–99.8% profiling reduction. Monotonicity of accuracy along prefixes is an assumption used for pruning and fill-in; if false it is a correctness risk, not a circular reduction of the claimed measurements to their inputs. The derivation chain is therefore self-contained against external baselines and does not exhibit the enumerated circularity patterns.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 1 invented entities

The load-bearing claims rest on a small set of domain assumptions about workflow semantics and request statistics plus a handful of engineering knobs (profiling coverage, rank-1 smoothing). No new physical entities are postulated; the execution trie is a data-structure representation, not an ontological invention.

free parameters (3)
  • sparse profiling coverage = 2% (main results)
    Fraction of exhaustive LLM cost used for cascade sampling (reported at 0.2–2%); chosen by the authors and shown to be sufficient for ~1% MAE, but still a free engineering knob that affects estimator variance.
  • rank-1 SVD projection on depth-3 conditional matrix = rank 1
    Variance-reduction step applied only to the sparsest block; the rank is chosen by hand rather than cross-validated.
  • maximum retry / reflection horizon = workflow-specific (3–6)
    Bounds the finite trie; set per workflow (depth 3–6) by the authors to keep the path set tractable.
axioms (4)
  • domain assumption Accuracy is monotone non-decreasing along every root-to-leaf path because extra stages only add refinement opportunities.
    Stated in §3.3 and used for pruning; if a later stage can corrupt a correct intermediate answer the monotonicity (and therefore pruning safety) fails.
  • domain assumption The offline request set Q is large and drawn from a stationary distribution so that column means transfer to online traffic.
    Invoked via the law of large numbers in §3.4; distribution shift would invalidate the annotated trie.
  • domain assumption Success is prefix-closed: if any prefix succeeds then every extension is also counted successful.
    Enables subtree fill-in and the cascade decomposition equations (1) and (4)–(6) in the appendix.
  • standard math Standard matrix-completion and low-rank SVD machinery apply to the conditional accuracy matrix after cascade correction.
    Used only as a variance-reduction post-process on the already-unbiased conditional estimates.
invented entities (1)
  • annotated execution trie of model-choice prefixes independent evidence
    purpose: Compact representation of all feasible per-invocation plans that exposes shared prefixes for checkpointing and online re-rooting.
    The central data structure of the paper; it is a systems abstraction rather than a new physical or mathematical object, and its utility is demonstrated empirically rather than postulated.

pith-pipeline@v1.1.0-grok45 · 26608 in / 2859 out tokens · 33775 ms · 2026-07-12T23:55:01.429494+00:00 · methodology

0 comments
read the original abstract

Agentic workflows interleave configurable LLM stages with tool stages and often include retries or refinement loops. Existing workflow managers profile full workflow configurations offline and assign each request a static workflow-level plan that binds each configurable LLM stage to a single model, reuses that model across repeated loop iterations, and does not revisit those choices at runtime. We present VineLM, a workflow manager that enables fine-grained control by choosing the model for each stage invocation as execution unfolds under request-level objectives such as maximizing accuracy under cost or latency budgets. VineLM represents feasible executions as an annotated trie of model-choice prefixes and uses checkpointing and cascade profiling to estimate path accuracy, cost, and latency without exhaustively profiling every request on every path. At runtime, VineLM re-roots the trie after each stage invocation and replans over the remaining subtrie using the realized execution prefix and remaining latency budget. On NL2SQL and math reasoning workflows, VineLM improves the cost-latency-accuracy frontier over coarse workflow-level baselines, achieving up to 18% higher accuracy at the same per-request budget with its sparse profiling reducing offline profiling cost by 98-99.8% when compared to exhaustive profiling.

Figures

Figures reproduced from arXiv: 2605.23914 by Dan Rubenstein, Kostis Kaffes, Matthew Lou, Nikos Pagonas, Tianyi Peng.

Figure 1
Figure 1. Figure 1: A simplified example NL2SQL workflow contain￾ing a SQL Generation LLM stage (G), a SQL Execution tool stage (E), and a potentially repeated Repair LLM stage (R). Existing workflow managers such as Murakkab [9] commit to one workflow-level model assignment when the request arrives, with repeated loop iterations always reusing the same model. VineLM instead reconsiders the choice after each stage and can mix… view at source ↗
Figure 2
Figure 2. Figure 2: shows why this matters on a concrete NL2SQL request with a fixed cost SLO. Static single-model strategies fail in opposite ways: always using GLM 4.7 reaches the correct answer but exceeds the budget, while always using Gemma 27B or always using Qwen 32B stays within budget but returns the wrong answer. The successful path is mixed: GLM 4.7 for the initial generation, then Gemma 27B and Qwen 32B on success… view at source ↗
Figure 3
Figure 3. Figure 3: For this MathQA request with three stages, the ob￾jective is to maximize accuracy while meeting a 15 s latency SLO. Offline averages favor the plan Gemma 27B → Sonnet 4.6 → Sonnet 4.6, but the realized second step runs long, so following that plan finishes in 16.7 s and violates the SLO (orange). Replanning after step 2 and switching the final step to Gemma 27B yields Gemma 27B → Sonnet 4.6 → Gemma 27B, fi… view at source ↗
Figure 5
Figure 5. Figure 5: The request–path accuracy table 𝐴 under 5% cas￾cade sampling. Columns are sorted left to right by depth: 8 depth-1 paths, 64 depth-2 paths, and 512 depth-3 paths (separated by vertical lines). Missingness is severe and depth￾dependent: depth-1 columns are 97% observed, depth-2 only 17%, and depth-3 just 1.8%, because later stages are only reached when earlier stages are executed. Each color indi￾cates a de… view at source ↗
Figure 6
Figure 6. Figure 6: VineLM overview. At runtime, each incoming request arrives together with an objective 𝑜 = (𝑓 , C), such as minimizing cost subject to an accuracy floor and latency cap. The VineLM controller begins at the root of the trie and repeatedly chooses the next stage/model decision based on the current execution prefix, the remaining budget implied by 𝑜, and the estimated metrics stored in the trie. The chosen act… view at source ↗
Figure 7
Figure 7. Figure 7: Accuracy delta over Murakkab for three workflows: NL2SQL-8, NL2SQL-2, and MathQA. MathQA shows a smaller but still positive delta for a differ￾ent reason. Because MathQA is a repeated-reflection work￾flow, Murakkab effectively commits to one model for the reflection process, whereas VineLM can mix models across rounds. The gain is smaller not because fine-grained con￾trol stops helping, but because baselin… view at source ↗
Figure 8
Figure 8. Figure 8: Column-mean prediction error versus profiling coverage. Coverage is the fraction of full offline LLM profiling cost used for sparse sampling [PITH_FULL_IMAGE:figures/full_fig_p010_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Maximum accuracy under a cost SLO (left) and minimum average cost under an accuracy SLO (right), at 2% profiling coverage. error separates pessimism from optimism. The direct average is strongly pessimistic because direct averaging observes only the suffix-conditioned hard subpopulation. Prefix-based methods are optimistic on average. VineLM variants are nearly unbiased in mean signed error, which shows th… view at source ↗
Figure 10
Figure 10. Figure 10: Latency-SLO violation rate for Murakkab, dynamic load-unaware control, and dynamic load-aware control. Dynamic replanning reduces violations, and load-aware replanning reduces them further under load [PITH_FULL_IMAGE:figures/full_fig_p012_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

63 extracted references · 4 linked inside Pith

  1. [1]

    Amazon Bedrock, 2026

    Amazon Web Services. Amazon Bedrock, 2026. Accessed: 2026-04-02

    2026

  2. [2]

    Amazon Elastic Compute Cloud (Amazon EC2), 2026

    Amazon Web Services. Amazon Elastic Compute Cloud (Amazon EC2), 2026. Accessed: 2026-04-02

    2026

  3. [3]

    Strands agents — open source ai agent sdk for python & typescript.https://strandsagents.com/, 2026

    Amazon Web Services, Inc. Strands agents — open source ai agent sdk for python & typescript.https://strandsagents.com/, 2026. Accessed: 2026-04-02

    2026

  4. [4]

    Claude Sonnet, 2024

    Anthropic. Claude Sonnet, 2024. Accessed: 2026-04-02

    2024

  5. [5]

    Agent sdk overview - claude api docs.https://platform

    Anthropic. Agent sdk overview - claude api docs.https://platform. claude.com/docs/en/agent-sdk/overview, 2026. Accessed: 2026-04-02

    2026

  6. [6]

    Graph of thoughts: Solving elaborate problems with large language mod- els.Proceedings of the AAAI Conference on Artificial Intelligence, 38(16):17682–17690, March 2024

    Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hu- bert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler. Graph of thoughts: Solving elaborate problems with large language mod- els.Proceedings of the AAAI Conference on Artificial Intelligence, 38(16):17682–17690, March 2024

    2024

  7. [7]

    Prompting is programming: A query language for large language models.Proc

    Luca Beurer-Kellner, Marc Fischer, and Martin Vechev. Prompting is programming: A query language for large language models.Proc. ACM Program. Lang., 7(PLDI), June 2023

    2023

  8. [8]

    Candès and Benjamin Recht

    Emmanuel J. Candès and Benjamin Recht. Exact matrix completion via convex optimization.Foundations of Computational Mathematics, 9(6):717–772, 2009

    2009

  9. [9]

    Murakkab: Resource-efficient agentic workflow orchestration in cloud platforms, 2025

    Gohar Irfan Chaudhry, Esha Choukse, Haoran Qiu, Íñigo Goiri, Ro- drigo Fonseca, Adam Belay, and Ricardo Bianchini. Murakkab: Resource-efficient agentic workflow orchestration in cloud platforms, 2025

    2025

  10. [10]

    FrugalGPT: How to use large language models while reducing cost and improving perfor- mance.Transactions on Machine Learning Research, 2024

    Lingjiao Chen, Matei Zaharia, and James Zou. FrugalGPT: How to use large language models while reducing cost and improving perfor- mance.Transactions on Machine Learning Research, 2024. Featured Certification

    2024

  11. [11]

    Xgboost: A scalable tree boosting system

    Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. InProceedings of the 22nd ACM SIGKDD International Confer- ence on Knowledge Discovery and Data Mining, KDD ’16, page 785–794, New York, NY, USA, 2016. Association for Computing Machinery

    2016

  12. [12]

    Do large language models need a content delivery network?, 2024

    Yihua Cheng, Kuntai Du, Jiayi Yao, and Junchen Jiang. Do large language models need a content delivery network?, 2024

    2024

  13. [13]

    Kakkar, Yu Gan, Brenton Milne, and Fatma Ozcan

    Yeounoh Chung, Gaurav T. Kakkar, Yu Gan, Brenton Milne, and Fatma Ozcan. Is long context all you need? leveraging LLM’s extended context for NL2SQL.arXiv preprint arXiv:2501.12372, 2025

    Pith/arXiv arXiv 2025

  14. [14]

    Inferline: latency- aware provisioning and scaling for prediction serving pipelines

    Daniel Crankshaw, Gur-Eyal Sela, Xiangxi Mo, Corey Zumar, Ion Stoica, Joseph Gonzalez, and Alexey Tumanov. Inferline: latency- aware provisioning and scaling for prediction serving pipelines. In Proceedings of the 11th ACM Symposium on Cloud Computing, SoCC ’20, page 477–491, New York, NY, USA, 2020. Association for Computing Machinery

    2020

  15. [15]

    Franklin, Joseph E

    Daniel Crankshaw, Xin Wang, Guilio Zhou, Michael J. Franklin, Joseph E. Gonzalez, and Ion Stoica. Clipper: A Low-Latency online prediction serving system. In14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17), pages 613–627, Boston, MA, March 2017. USENIX Association

    2017

  16. [16]

    Crewai: The leading multi-agent platform.https://crewai

    CrewAI. Crewai: The leading multi-agent platform.https://crewai. com/, 2026. Accessed: 2026-04-02

    2026

  17. [17]

    Deepseek-v3.2: Pushing the frontier of open large lan- guage models, 2025

    DeepSeek-AI. Deepseek-v3.2: Pushing the frontier of open large lan- guage models, 2025

    2025

  18. [18]

    Jasper Dekoninck, Maximilian Baader, and Martin T. Vechev. A unified approach to routing and cascading for llms. InICML, 2025

    2025

  19. [20]

    Amdahl’s law for tail latency.Commun

    Christina Delimitrou and Christos Kozyrakis. Amdahl’s law for tail latency.Commun. ACM, 61(8):65–72, July 2018

    2018

  20. [21]

    Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Ruhle, Laks V. S. Lakshmanan, and Ahmed Has- san Awadallah. Hybrid llm: Cost-efficient and quality-aware query routing, 2024

    2024

  21. [22]

    gemma-3-27b-it Model Card, 2025

    Google. gemma-3-27b-it Model Card, 2025. Accessed: 2026-04-02

    2025

  22. [23]

    Matrix completion and low-rank svd via fast alternating least squares, 2014

    Trevor Hastie, Rahul Mazumder, Jason Lee, and Reza Zadeh. Matrix completion and low-rank svd via fast alternating least squares, 2014

    2014

  23. [24]

    MetaGPT: Meta programming for a multi- agent collaborative framework

    Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. MetaGPT: Meta programming for a multi- agent collaborative framework. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  24. [25]

    Dspy: Compiling declarative language model calls into self-improving pipelines.arXiv preprint arXiv:2310.03714, 2023

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, et al. Dspy: Compiling declarative language model calls into self-improving pipelines.arXiv preprint arXiv:2310.03714, 2023

    Pith/arXiv arXiv 2023

  25. [26]

    Matrix factorization techniques for recommender systems.Computer, 42(8):30–37, 2009

    Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recommender systems.Computer, 42(8):30–37, 2009

    2009

  26. [27]

    Langgraph, 2024

    LangChain, Inc. Langgraph, 2024

    2024

  27. [28]

    Lewis, Sebastián Echeverría, Lena Pons, and Jeffrey Chrabaszcz

    Grace A. Lewis, Sebastián Echeverría, Lena Pons, and Jeffrey Chrabaszcz. Augur: a step towards realistic drift detection in pro- duction ml systems. InProceedings of the 1st Workshop on Software Engineering for Responsible AI, SE4RAI ’22, page 37–44, New York, NY, USA, 2023. Association for Computing Machinery

    2023

  28. [29]

    LlamaIndex, 11 2022

    Jerry Liu. LlamaIndex, 11 2022

    2022

  29. [30]

    Lmcache: An efficient kv cache layer for enterprise- scale llm inference, 2025

    Yuhan Liu, Yihua Cheng, Jiayi Yao, Yuwei An, Xiaokun Chen, Shaot- ing Feng, Yuyang Huang, Samuel Shen, Rui Zhang, Kuntai Du, and Junchen Jiang. Lmcache: An efficient kv cache layer for enterprise- scale llm inference, 2025

    2025

  30. [31]

    Cachegen: Kv cache compression and streaming for fast large language model serving

    Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Anantha- narayanan, Michael Maire, Henry Hoffmann, Ari Holtzman, and Junchen Jiang. Cachegen: Kv cache compression and streaming for fast large language model serving. InProceedings of the ACM SIG- COMM 2024 Conference, ACM SIGCOMM ’24, pag...

    2024

  31. [32]

    Gonzalez, and Ion Stoica

    Michael Luo, Xiaoxiang Shi, Colin Cai, Tianjun Zhang, Justin Wong, Yichuan Wang, Chi Wang, Yanping Huang, Zhifeng Chen, Joseph E. Gonzalez, and Ion Stoica. Autellix: An efficient serving engine for llm agents as general programs, 2025

    2025

  32. [33]

    Self-refine: Iterative refinement with self-feedback, 2023

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback, 2023

    2023

  33. [34]

    Match- maker: Data drift mitigation in machine learning for large-scale sys- tems

    Ankur Mallick, Kevin Hsieh, Behnaz Arzani, and Gauri Joshi. Match- maker: Data drift mitigation in machine learning for large-scale sys- tems. In D. Marculescu, Y. Chi, and C. Wu, editors,Proceedings of Machine Learning and Systems, volume 4, pages 77–94, 2022

    2022

  34. [35]

    D. Q. Mayne, J. B. Rawlings, C. V. Rao, and P. O. M. Scokaert. Sur- vey constrained model predictive control: Stability and optimality. Automatica, 36(6):789–814, June 2000

    2000

  35. [36]

    Llama-3.3-70B-Instruct Model Card, 2024

    Meta. Llama-3.3-70B-Instruct Model Card, 2024. Accessed: 2026-04-02

    2024

  36. [37]

    Gonzalez, M Waleed Kadous, and Ion Stoica

    Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, and Ion Stoica. RouteLLM: Learning to route LLMs from preference data. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  37. [38]

    gpt-oss-120b & gpt-oss-20b Model Card, 2025

    OpenAI. gpt-oss-120b & gpt-oss-20b Model Card, 2025. Accessed: 2026-04-02

    2025

  38. [39]

    Openai agents sdk.https://openai.github.io/openai-agents- python/, 2026

    OpenAI. Openai agents sdk.https://openai.github.io/openai-agents- python/, 2026. Accessed: 2026-04-02

    2026

  39. [40]

    Kvflow: Efficient prefix caching for accelerating llm-based multi-agent workflows.arXiv preprint arXiv:2507.07400, 2025

    Zaifeng Pan, Ajjkumar Patel, Zhengding Hu, Yipeng Shen, Yue Guan, Wan-Lu Li, Lianhui Qin, Yida Wang, and Yufei Ding. Kvflow: Efficient prefix caching for accelerating llm-based multi-agent workflows.arXiv preprint arXiv:2507.07400, 2025. 13

    Pith/arXiv arXiv 2025

  40. [41]

    Patil, Tianjun Zhang, Xin Wang, and Joseph E

    Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive apis, 2023

    2023

  41. [42]

    Toolllm: Facilitating large language models to master 16000+ real-world apis, 2023

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models to master 16000+ real-world apis, 2023

    2023

  42. [43]

    Self-reflection in LLM agents: Ef- fects on problem-solving performance.arXiv preprint arXiv:2405.06682, 2024

    Matthew Renze and Erhan Guven. Self-reflection in LLM agents: Ef- fects on problem-solving performance.arXiv preprint arXiv:2405.06682, 2024

    Pith/arXiv arXiv 2024

  43. [44]

    Yadwadkar, and Christos Kozyrakis

    Francisco Romero, Qian Li, Neeraja J. Yadwadkar, and Christos Kozyrakis. INFaaS: Automated model-less inference serving. In2021 USENIX Annual Technical Conference (USENIX ATC 21), pages 397–411. USENIX Association, July 2021

    2021

  44. [45]

    Yadwadkar, and Christos Kozyrakis

    Francisco Romero, Mark Zhao, Neeraja J. Yadwadkar, and Christos Kozyrakis. Llama: A heterogeneous & serverless framework for auto- tuning video analytics pipelines. InProceedings of the ACM Symposium on Cloud Computing, SoCC ’21, page 1–17, New York, NY, USA, 2021. Association for Computing Machinery

    2021

  45. [46]

    Toolformer: Language models can teach themselves to use tools

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. InThirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  46. [47]

    Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face, 2023

    Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face, 2023

    2023

  47. [48]

    Agentxray: White-boxing agentic systems via workflow reconstruction, 2026

    Ruijie Shi, Houbin Zhang, Yuecheng Han, Yuheng Wang, Jingru Fan, Runde Yang, Yufan Dang, Huatao Li, Dewen Liu, Yuan Cheng, and Chen Qian. Agentxray: White-boxing agentic systems via workflow reconstruction, 2026

    2026

  48. [49]

    Reflexion: Language agents with verbal reinforcement learning, 2023

    Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning, 2023

    2023

  49. [50]

    Irt-router: Effective and inter- pretable multi-llm routing via item response theory, 2025

    Wei Song, Zhenya Huang, Cheng Cheng, Weibo Gao, Bihan Xu, Guan- Hao Zhao, Fei Wang, and Runze Wu. Irt-router: Effective and inter- pretable multi-llm routing via item response theory, 2025

    2025

  50. [51]

    Kimi k2.5: Visual agentic intelligence, 2026

    Kimi Team. Kimi k2.5: Visual agentic intelligence, 2026

    2026

  51. [52]

    Causal LLM routing: End-to-end regret minimization from observational data

    Asterios Tsiourvas, Wei Sun, and Georgia Perakis. Causal LLM routing: End-to-end regret minimization from observational data. InThe Thirty- ninth Annual Conference on Neural Information Processing Systems, 2025

    2025

  52. [53]

    Efficient llm serving for agentic workflows: A data systems perspective, 2026

    Noppanat Wadlom, Junyi Shen, and Yao Lu. Efficient llm serving for agentic workflows: A data systems perspective, 2026

    2026

  53. [54]

    A survey on large language model based autonomous agents.Frontiers of Computer Science, 18(6), March 2024

    Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Jirong Wen. A survey on large language model based autonomous agents.Frontiers of Computer Science, 18(6), March 2024

    2024

  54. [55]

    Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models, 2023

    Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models, 2023

    2023

  55. [56]

    White, Doug Burger, and Chi Wang

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Eric Zhu, Li Jiang, Shaokun Zhang, Xiaoyun Zhang, Jiale Liu, Ahmed Has- san Awadallah, Ryen W. White, Doug Burger, and Chi Wang. Autogen: Enabling next-gen LLM applications via multi-agent conversation framework, 2023

    2023

  56. [57]

    Rewoo: Decoupling reasoning from observa- tions for efficient augmented language models, 2023

    Binfeng Xu, Zhiyuan Peng, Bowen Lei, Subhabrata Mukherjee, Yuchen Liu, and Dongkuan Xu. Rewoo: Decoupling reasoning from observa- tions for efficient augmented language models, 2023

    2023

  57. [58]

    Qwen3 technical report, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, ...

    2025

  58. [59]

    Cacheblend: Fast large language model serving for rag with cached knowledge fusion

    Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. Cacheblend: Fast large language model serving for rag with cached knowledge fusion. In Proceedings of the Twentieth European Conference on Computer Systems, EuroSys ’25, page 94–109, New York, NY, USA, 2025. Association for Computing Machinery

    2025

  59. [60]

    Griffiths, Yuan Cao, and Karthik Narasimhan

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023

    2023

  60. [61]

    React: Synergizing reasoning and acting in language models, 2023

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2023

    2023

  61. [62]

    AFlow: Automating agentic workflow generation

    Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xiong-Hui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, Bingnan Zheng, Bang Liu, Yuyu Luo, and Chenglin Wu. AFlow: Automating agentic workflow generation. InThe Thirteenth Interna- tional Conference on Learning Representations, 2025

    2025

  62. [63]

    Gonzalez, Clark Barrett, and Ying Sheng

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. Sglang: efficient execution of structured language model programs. InProceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, N...

    2024

  63. [64]

    GLM-4.7 Technical Blog, 2025

    Zhipu AI. GLM-4.7 Technical Blog, 2025. Accessed: 2026-04-02. 14 A Estimating Accuracy for Workflow Instances under MNAR Sampling This appendix describes VineLM’s approach to estimating the expected accuracy of every workflow instance from sparse offline profiling. We show that the cascade structure of the workflow tree, which at first appears to make the...

    2025