A new fault-injection framework enables a systematic empirical study that produces 17 takeaways on error propagation in LLM inference and four software-only mitigation directions.
Inference economics of language models
8 Pith papers cite this work. Polarity classification is still indexing.
8
Pith papers citing it
abstract
We develop a theoretical model that addresses the economic trade-off between cost per token versus serial token generation speed when deploying LLMs for inference at scale. Our model takes into account arithmetic, memory bandwidth, network bandwidth and latency constraints; and optimizes over different parallelism setups and batch sizes to find the ones that optimize serial inference speed at a given cost per token. We use the model to compute Pareto frontiers of serial speed versus cost per token for popular language models.
citation-role summary
background 3
citation-polarity summary
roles
background 3polarities
background 3representative citing papers
Simulation study shows cold TLB misses in reverse address translation dominate latency for small collectives in multi-GPU pods, causing up to 1.4x degradation, while larger ones see diminishing returns.
LLM serving should triage by five-resource analytical floors and wall ordering, not grid search; on 16×H20, TP16 is capacity-capped at ~70 while EP+DP attention reaches ~644 concurrent 8K requests.
Effective LLM inference cost per million output tokens varies 2.5-36x with offered request rate due to utilization, addressed by a concurrency-aware measurement methodology and open-source vLLM tool validated across model types.
The Productivity-Reliability Paradox arises because AI code generators produce variable output while developers lack sufficient specification discipline, making governance models focused on specifications the binding constraint rather than model improvements.
Techno-economic framework shows that GPU AI-RAN deployments can offset extra costs via AI revenue for up to 8x ROI across scenarios with varying token depreciation, demand, and GPU densities.
Post-training N:M activation pruning preserves generative performance in LLMs better than equivalent weight pruning, with the 8:16 pattern emerging as a practical hardware-friendly choice.
Tokalator is a toolkit with VS Code extension, calculators, and community resources to monitor and optimize token usage in AI coding environments.
citing papers explorer
-
Not All Errors Are Equal: A Systematic Study of Error Propagation in Large Language Model Inference
A new fault-injection framework enables a systematic empirical study that produces 17 takeaways on error propagation in LLM inference and four software-only mitigation directions.
-
Analyzing Reverse Address Translation Overheads in Multi-GPU Scale-Up Pods
Simulation study shows cold TLB misses in reverse address translation dominate latency for small collectives in multi-GPU pods, causing up to 1.4x degradation, while larger ones see diminishing returns.
-
Think Before You Grid-Search: Floor-First Triage for LLM Serving
LLM serving should triage by five-resource analytical floors and wall ordering, not grid search; on 16×H20, TP16 is capacity-capped at ~70 while EP+DP attention reaches ~644 concurrent 8K requests.
-
Beyond Per-Token Pricing: A Concurrency-Aware Methodology for LLM Infrastructure Cost Estimation
Effective LLM inference cost per million output tokens varies 2.5-36x with offered request rate due to utilization, addressed by a concurrency-aware measurement methodology and open-source vLLM tool validated across model types.
-
The Productivity-Reliability Paradox: Specification-Driven Governance for AI-Augmented Software Development
The Productivity-Reliability Paradox arises because AI code generators produce variable output while developers lack sufficient specification discipline, making governance models focused on specifications the binding constraint rather than model improvements.
-
A Techno-Economic Framework for Cost Modeling and Revenue Opportunities in Open and Programmable AI-RAN
Techno-economic framework shows that GPU AI-RAN deployments can offset extra costs via AI revenue for up to 8x ROI across scenarios with varying token depreciation, demand, and GPU densities.
-
Motivating Next-Gen Accelerators with Flexible (N:M) Activation Sparsity via Benchmarking Lightweight Post-Training Sparsification Approaches
Post-training N:M activation pruning preserves generative performance in LLMs better than equivalent weight pruning, with the 8:16 pattern emerging as a practical hardware-friendly choice.
-
Tokalator: A Context Engineering Toolkit for Artificial Intelligence Coding Assistants
Tokalator is a toolkit with VS Code extension, calculators, and community resources to monitor and optimize token usage in AI coding environments.