REVIEW 21 cited by
Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoning
read the original abstract
Large language models (LLMs) such as GPT-4 have exhibited remarkable performance in a variety of tasks, but this strong performance often comes with the high expense of using paid API services. In this paper, we are motivated to study building an LLM cascade to save the cost of using LLMs, particularly for performing reasoning (e.g., mathematical, causal) tasks. Our cascade pipeline follows the intuition that simpler questions can be addressed by a weaker but more affordable LLM, whereas only the challenging questions necessitate the stronger and more expensive LLM. To realize this decision-making, we consider the "answer consistency" of the weaker LLM as a signal of the question difficulty and propose several methods for the answer sampling and consistency checking, including one leveraging a mixture of two thought representations (i.e., Chain-of-Thought and Program-of-Thought). Through experiments on six reasoning benchmark datasets, with GPT-3.5-turbo and GPT-4 being the weaker and stronger LLMs, respectively, we demonstrate that our proposed LLM cascades can achieve performance comparable to using solely the stronger LLM but require only 40% of its cost.
Forward citations
Cited by 21 Pith papers
-
Online Learning-to-Defer with Varying Experts
Presents the first online learning-to-defer algorithm with regret bounds O((n + n_e) T^{2/3}) generally and O((n + n_e) sqrt(T)) under low noise for multiclass classification with varying experts.
-
RouteJudge: An Open Platform for Reproducible and Preference-Aware LLM Routing
RouteJudge introduces an open platform for preference-based evaluation of LLM routers via pairwise user comparisons, along with the ORBIT toolbox for standardized routing workflows.
-
Online Learning-to-Defer with Varying Experts
Presents first online L2D algorithm for multiclass classification with bandit feedback and varying experts, achieving O((n+n_e)T^{2/3}) regret generally and O((n+n_e)√T) under low noise.
-
Online Learning-to-Defer with Varying Experts
Presents the first online Learning-to-Defer algorithm achieving regret O((n + n_e) T^{2/3}) generally and O((n + n_e) sqrt(T)) under low noise for multiclass classification with varying experts.
-
Conformal Cascade: Distribution-Free Accuracy Guarantees for Multi-Tier LLM Inference
Using conformal prediction-set size as the cascade deferral rule yields distribution-free cascade accuracy bounds and usually beats confidence-threshold heuristics on multiple-choice LLM benchmarks.
-
Conformal Cascade: Distribution-Free Accuracy Guarantees for Multi-Tier LLM Inference
A conformal-prediction set-size deferral rule gives LLM cascades a distribution-free marginal coverage bound of 1−Kα, with empirical gains over tuned heuristics reported at test-set-oracle operating points.
-
HACO: Hedged Agent Computing for Reliable LLM Systems
HACO adaptively selects a reliability-constrained hedge set of candidate LLM agent instances for each role invocation, improving robustness under changing deployment conditions at lower token/latency cost than full-po...
-
RLM-Cascade: Response-Level Speculative Decoding for Cost-Efficient LLM API Serving
RLM-Cascade applies response-level speculative decoding with a complexity router to reduce LLM API costs by 45.8% on agentic coding tasks while also lowering latency and matching or exceeding baseline quality.
-
SEMA-SQL: Beyond Traditional Relational Querying with Large Language Models
SEMA-SQL automates natural language to efficient hybrid queries combining relational algebra with LLM semantic operations via a new Hybrid Relational Algebra abstraction.
-
SEMA-SQL: Beyond Traditional Relational Querying with Large Language Models
SEMA-SQL formalizes Hybrid Relational Algebra to let users pose natural language questions answered by automatically generated queries that combine relational operators with LLM semantic reasoning, cutting LLM calls b...
-
Privacy-Preserving LLMs Routing
PPRoute achieves plaintext-level LLM routing quality with MPC-based privacy and a 20x speedup over naive encrypted implementations via MPC-friendly encoders, multi-step training, and O(1) communication Top-k search.
-
Agent-GWO: Collaborative Agents for Dynamic Prompt Optimization in Large Language Models
Agent-GWO uses collaborative grey-wolf-inspired agents to jointly optimize LLM prompts and decoding settings, yielding higher accuracy and stability than prior single-agent prompt optimization methods on math and hybr...
-
Ensemble-Based Uncertainty Estimation for Code Correctness Estimation
Ensemble Semantic Entropy improves correlation with code correctness over single-model methods and powers a cascading scaling system that cuts FLOPs by 64.9% while preserving performance on LiveCodeBench.
-
Think Short, Defer Smart, Act, and Repeat: Calibrated Reasoning and Uncertainty-Aware Deferral for Edge LLM Agents
Jointly calibrated early thought-stopping and perplexity deferral cut edge-agent thinking compute 43–73% while keeping finite-sample reward and cloud-call guarantees.
-
How Often Should a Recommender Call an LLM? Value-Weighted Routing, Monitoring, and Seasonal Robustness
Value-weighted LLM routing matches difficulty-only recall while raising precision, exposes within-category calibration collapse, and an elastic value-scaled budget absorbs a synthetic Black Friday surge.
-
From Sampled Outcomes to Capability Distributions: Rethinking Supervision for LLM Routing
DARS replaces single-shot response labels with distribution-aware supervision derived from input and output uncertainty to produce more reliable LLM routing policies.
-
Evaluating Small Language Models for Front-Door Routing: A Harmonized Benchmark and Synthetic-Traffic Experiment
Qwen-2.5-3B achieves 0.793 accuracy and 988 ms median latency on six-class task routing but misses the pre-registered viability bar of 0.85 accuracy and 2000 ms P95 latency.
-
Training-Free versus Training-Based Intent Classification in LLMs: Accuracy, Robustness, and Failure Modes
Statistical classifiers built on LLM activation norms and coordinates match or beat trained MLP heads on coarse intent routing and resist camouflage better, while MLPs win on fine-grained subfield distinctions.
-
Optimal Query Allocation in Extractive QA with LLMs: A Learning-to-Defer Framework with Theoretical Guarantees
A learning-to-defer framework allocates extractive QA queries to LLM experts with theoretical optimality guarantees, shown to improve reliability and cut overhead on SQuAD and TriviaQA.
-
Token-Operations-Oriented Inference Optimization Techniques for Large Models
A survey of large-model inference optimization, organized as a four-layer 'token-operations' taxonomy: multi-model fusion, model optimization, compute-model fusion, and compute-network-model fusion.
-
Token-Operations-Oriented Inference Optimization Techniques for Large Models
The paper introduces a four-layer technical architecture for token-operations-oriented inference optimization in large models and reviews key technologies and industry status at each layer.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.