GORGO: Online Tuning for Cross-Region Network-Aware LLM Serving

Abinaya Dinesh; Alessio Ricci Toniolo; Rome Thorstenson

REVIEW 4 major objections 6 minor 1 cited by

GORGO: Online Tuning for Cross-Region Network-Aware LLM Serving

T0 review · 4 major / 6 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Routing LLM requests by estimating time-to-first-token as the sum of forwarding latency, residual prefill, and queue delay cuts p95 tail latency by up to 31%.

desk verdict The abstract and the body report different methods and different headline results; the body's own numbers don't hold together, so treat the 2.5x claim as an interesting hypothesis, not a measured fact. read the letter →

arxiv 2602.11688 v3 pith:CRPQAYB2 submitted 2026-02-12 cs.NI cs.DC

Alessio Ricci Toniolo , Rome Thorstenson , Abinaya Dinesh This is my paper

classification cs.NIcs.DC

keywords LLMservingloadbalancingKV-cachereuseprefixcachinggeo-distributedinferencetime-to-first-tokencostmodelonlinetuning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

GORGO sets out to prove that request routing for geographically distributed LLM serving should minimize an estimated time-to-first-token that adds three measurable components: wide-area forwarding latency, residual prefill time after prefix-cache reuse, and admission/queue delay. The paper builds a load balancer around this additive cost model with tunable weights, tunes the weights with evolutionary strategies on a synthetic long-context chat dataset (ART-Chat-2.5M) so that p95 TTFT is optimized directly, then fixes the weights and evaluates on held-out traffic. It reports 6.9-15.5% lower p95 TTFT and 14.3-30.9% lower p95 end-to-end latency against baselines that chase only cache overlap or only load; a centralized proxy variant reports a 2.5x reduction in median TTFT. The payoff, if the result holds, is a practical answer to the question prior systems treated as either-or: when is a distant cache hit worth more than a nearby miss?

What carries the argument

The load-bearing object is the additive cost model that makes the three signals commensurable in milliseconds: a measured per-token prefill rate t_p (fitted as y = 150.72 + 0.0938x from 87 sample prompts, R² = 0.9863), a lightweight prefix trie mirroring the serving runtime's radix trie to estimate cached-prefix length L_hit, and measured RTT plus mirrored queue depth. Subtracting L_hit · t_p from total prefill length gives ResidualPrefillTime, so the router can compare a distant 15% cache hit against a local 5% hit after adding forwarding and queueing costs.

What would settle it

Re-run the three-region benchmark with a workload whose batch composition alternates between short and very long prompts and recompute the marginal prefill time per token for each batch; if the fixed t_p differs from the batch-observed rate by more than about 20%, the GORGO route choices will no longer match the empirically fastest region, and the claimed p95 TTFT improvements should shrink or reverse.

Watch

Extended reading notes

Core claim

The central claim is that a routing policy which prices network latency, prefix-cache reuse, and queue state in the same millisecond units, and whose two weights are tuned on a representative workload, beats policies that optimize any single one of these signals. Formally, the per-region cost is Cost(region) = NetworkLatency(peer) + t_p · PrefillCost(peer + local) + q̂_s · QueueWaitTime(peer), where t_p = 0.0938 ms/token converts the cached-prefix length into saved prefill time, and q̂_s weights admission delay. The paper argues this additive form is justified because the components occur sequentially in the serving pipeline, and demonstrates that the centralized GORGO-proxy, which has fresh

Load-bearing premise

The routing decisions assume the per-token prefill rate measured once from 87 sample prompts stays constant across all prompt lengths, batch compositions, and load levels, and that the load balancer's simplified prefix tree matches the serving runtime's true cache state; if either drifts, the cache term misprices routes and GORGO reduces to network-plus-queue routing.

Editorial extensions

If this is right

If correct, multi-region LLM gateways should stop treating inference as an opaque endpoint and instead expose prefix-overlap and admission state to the router.
Cache-aware routing that ignores wide-area latency can worsen tail TTFT; prefix locality should be valued by the compute time it actually saves.
A centralized proxy with fresh global state can beat distributed per-region balancers at this scale, because peer summaries drift.
The two tunable weights can be set once by evolutionary search on a representative dataset and fixed for held-out evaluation, so deployment does not require online learning.
The released synthetic long-context, high-prefix-reuse dataset (ART-Chat-2.5M) fills a gap in public chat traces used for benchmarking such policies.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the 2.5x median TTFT gap between the centralized proxy and the distributed balancer suggests information freshness, not the cost model itself, may be the dominant lever; a decentralized design with faster summary exchange might reclaim most of the gain at lower coordination cost.
Editorial inference: because t_p is fitted once from 87 samples, the model implicitly assumes prefill throughput is stable across batch composition; a natural robustness test is to re-fit t_p per batch or load level and check whether routing decisions change.
Editorial inference: the paper's own limitation (§7) admits the lightweight prefix index approximates the runtime's exact KV state; one could quantify this gap by logging true hit lengths and comparing, which would bound the model's mispricing error.
Editorial inference: the reported throughput penalty for the distributed policy (§7) suggests a joint TTFT-throughput objective could be tuned the same way, trading a small tail-latency loss for higher utilization.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick read on arXiv:2602.11688. The abstract promises evolutionary online tuning on a new ART-Chat-2.5M dataset with p95 improvements. The full text never mentions that dataset, that tuning strategy, or any p95 numbers. The body fits a per-token prefill rate from 87 WildChat samples and reports a 2.5x median TTFT reduction for a centralized proxy. That is a real mismatch, and it's the first thing anyone will notice.

Credit where it's due: the core idea is sound. Routing in geo-distributed LLM serving really is a three-way tradeoff between prefix reuse, network RTT, and admission/queue state, and GORGO formalizes that as an additive cost model. The centralized GORGO-proxy result—that a single coordinator with a full view beats decentralized per-region balancers—is a legitimate, non-obvious empirical claim, and the comparison against least-load and prefix-trie baselines is the right kind of experiment. If the data were trustworthy, the 2.5x median TTFT improvement would be a useful data point.

The problem is the data. The appendix tables are self-contradictory: median TTFT is 568 ms while median Request Latency is 7 ms, which is impossible if Request Latency includes TTFT plus decode. And Tokens/sec maxima in the millions on a single 8×A100 node serving Mistral-7B are not plausible—two orders of magnitude beyond what that hardware can do. These aren't cosmetic typos; they mean either the metric definitions or the measurement pipeline is unreliable. The t_p fit from 87 samples is thin but tolerable as a first-cut; the prefix-trie approximation is acknowledged in Section 7. But you can't build a valid head-to-head comparison on top of internally inconsistent measurements. The two code URLs also conflict, and there's no commit hash.

Net: the conceptual contribution is worth thinking with, but this draft is not self-consistent and not reproducible. I would not send it to peer review as is. The right move is to reject and invite a resubmission that aligns the abstract with the body, fixes the metric definitions, reports p95/E2E numbers if they exist, and reruns the experiments with sane instrumentation. As it stands, the paper's headline claim is unsupported.

Would I cite it? Not in the next year. But if you're reading this for the routing idea, skip the tables and read Sections 3 and 5.

Referee Report

4 major / 6 minor

Summary. The paper proposes GORGO, a per-region load-balancing proxy for geo-distributed LLM serving that routes each request to the region minimizing an additive estimate of TTFT: measured network RTT plus residual prefill time after prefix-cache reuse plus a weighted queueing delay. The policy is maintained with a lightweight prefix trie and per-node telemetry. The body evaluates GORGO and a centralized GORGO-proxy against least-load, prefix-trie, and GORGO baselines in a three-region (US-West, Germany, Israel) 8×A100 setup serving Mistral-7B-Instruct-v0.3 with GuideLLM/WildChat workloads, and claims a 2.5× median TTFT reduction for GORGO-proxy over least-load. The abstract additionally promises p95 TTFT/E2E improvements of 6.9–15.5% / 14.3–30.9% obtained by evolutionary tuning on a released ART-Chat-2.5M dataset with held-out evaluation; this material does not appear in the body. The appendix reports detailed per-policy metrics.

Significance. The routing objective—jointly treating cache reuse, network latency, and queueing admission as a tunable additive cost—is a reasonable and timely systems idea, and the centralized-proxy comparison point is worthwhile. The paper ships code and a dataset (GORGO on github, ART-Chat-2.5M), which is a strength, though I did not verify reproducibility. However, the paper as submitted does not establish the headline contributions. The abstract's p95 results from evolutionary tuning on ART-Chat-2.5M are absent from the body; the body's evaluation has internal metric inconsistencies that make even the median TTFT comparison unreliable; and the only statistical support is a single 60-second run with 57–141 requests per policy. If the claims were reproduced, the idea would be useful, but the current evidence is not sufficient.

major comments (4)

[Abstract vs. §4.2–§4.5] The abstract claims p95 TTFT improvements of 6.9–15.5% and p95 E2E latency improvements of 14.3–30.9% from evolutionary tuning on ART-Chat-2.5M with held-out evaluation windows. The body (§4.2–§4.5, §4.6) does not mention ART-Chat-2.5M, evolutionary strategies, held-out windows, or p95 E2E; instead, §4.5 reports a 2.5x reduction in median TTFT for GORGO-proxy based on single 60s runs. No experiment in the paper corresponds to the abstract's central claim, so the primary stated contribution is unsupported.
[Appendix A, Tables 2–5] In every table, median "Request Latency" (e.g., 7.07 ms for least-load) is an order of magnitude smaller than median TTFT (568.27 ms), which is impossible if Request Latency includes the time to first token; if Request Latency is only downstream processing, the metric is misnamed and uninformative. Additionally, max Tokens/sec values (2.78M, 7.38M, 9.09M, 7.22M) on a single 8×A100 node serving Mistral-7B are orders of magnitude above achievable throughput. These inconsistencies imply the measurement pipeline or metric definitions are unreliable, so the 2.5x median TTFT claim cannot be taken at face value.
[§4.6 and Tables 2–5] Each policy is benchmarked in a single 60-second run with 10 concurrent requests, yielding N=57–141 requests for TTFT. P95/P99 tail estimates from 57–141 samples have very wide confidence intervals, and the paper provides no error bars or repeated runs. The paper's own §7 says the evaluation is "constrained to a small number of regions and a limited set of deployment configurations." For a paper whose stated goal is reducing tail latency, this is insufficient evidence; the abstract's p95 numbers cannot be derived from this setup.
[§3.4 and §4.2] The cost model converts prefix overlap to time using a single per-token prefill rate t_p fitted on 87 WildChat samples (Table 1: y = 150.72 + 0.0938x, R² = 0.9863), then extrapolates to all prompt lengths and batch compositions. §7 concedes that "prefix-overlap estimates are derived from a lightweight prefix index rather than from the serving runtime's exact KV state, and per-token rates may shift with batch composition." Since ResidualPrefillTime = (L_p − L_hit)·t_p is the key cache-aware term, the paper needs a sensitivity analysis showing how errors in L_hit or t_p affect routing decisions; without it, the reported gains could vanish under realistic drift. The limitation is stated, but no mitigation or robustness experiment is provided.

minor comments (6)

[Abstract and §1 footnote] The code/data link in the Abstract (github.com/Arcadia-Research-Team/GORGO) differs from the footnote's link (github.com/atoniolo76/gotoni/tree/benchmark-load-balancing). Please reconcile these references.
[Table 1] Table 1 has no caption; the caption is necessary because the regression setup (model, hardware, batch size, etc.) is described only textually in §4.2.
[§3.4] The notation for the queue-weight parameter is inconsistent: §3.4 writes q̂_s but the text and §8 refer to q̂_s / tp; define the variables in one place.
[Figures 2–3] Figures 2 and 3 (median/mean latency and throughput across methods) are not described with error bars or confidence intervals; the single-run nature of the data should be stated in the figure captions.
[§4.3 and §4.6] The workload "GuideLLM" is used but is not included in the references; provide a citation or describe the tool.
[Abstract, §4.3, Tables 2–5] The paper uses "P99" in one abstract, "p95" in another abstract and in the tables, and §4.3 mentions measuring improvements at tail latencies (P99). Use consistent naming and define the reported percentile in the main text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the routing policy is a fitted surrogate evaluated by measured TTFT, not a derivation equivalent to its inputs.

full rationale

The central chain in GORGO is: define a per-region additive cost model (NetworkLatency + t_p*PrefillCost + q_s*QueueWaitTime), fit t_p from 87 WildChat samples, use the cost to route, and then measure TTFT against baselines. None of these steps reduces to its own inputs by construction. The cost model in §3.4 is explicitly a heuristic estimate, and §5 says: 'the model is not intended to perfectly predict per-request latency; rather, it operationalizes a consistent set of tradeoffs that can be evaluated and tuned within a deployment.' The reported improvements in §4.5 are measured TTFT values from the benchmarking runs, not algebraic consequences of the cost equation. The fitted t_p (Table 1: y = 150.72 + 0.0938x, R^2 = 0.9863) is a calibration parameter, not a prediction being validated; the paper does not claim the R^2 as held-out evidence. §7 concedes real approximation limits: 'prefix-overlap estimates are derived from a lightweight prefix index rather than from the serving runtime's exact KV state, and per-token rates may shift with batch composition.' Those are accuracy risks, not circularity. There are no load-bearing self-citations: the references are to external systems (SGLang, SkyWalker, Preble, etc.), and no uniqueness theorem or prior result by the same authors is invoked to force the design. The abstract's p95 numbers and ART-Chat-2.5M tuning do not appear in the body, and the appendix metrics are internally inconsistent (e.g., median TTFT of 224-568 ms alongside ~7 ms 'Request Latency'), but these are consistency/reproducibility defects outside the circularity rubric. Under the required standard of quote-and-reduce, no circular step is present.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim leans on five modeling assumptions and three fitted parameters. No invented entities are introduced — GORGO is a software artifact, not a postulated entity. The heaviest burden is the constant prefill-rate model and the approximate prefix-trie mirror, both conceded in §7.

free parameters (3)

Per-token prefill rate t_p = 0.0938 ms/token
OLS slope from 87 WildChat samples (Table 1); converts prefix-reuse token counts into time in the GORGO cost model (§3.4) and is fit on the same data family used for evaluation.
Queue weight q̂s = not reported
Tunable weight on QueueWaitTime in §3.4; the full text says it is a deployment parameter and defers auto-tuning to future work (§8), while the abstract claims evolutionary strategies tune policy parameters but reports no values.
TTFT linear-model intercept = 150.72 ms
Fitted intercept in the linear TTFT model y = 150.72 + 0.0938x (Table 1); part of the timing estimate underlying the prefill-rate calibration.

assumptions (5)

domain assumption Additive, sequential cost decomposition: TTFT ≈ NetworkLatency + t_p·PrefillCost + q̂s·QueueWaitTime
Equation in §3.4. Assumes the three delay components are independent and additive with no interaction (e.g., no effect of network jitter, prefill-decode overlap, or batching dynamics).
domain assumption Constant per-token prefill time t_p across prompt lengths, batch compositions, and load
Table 1 linear fit (R² = 0.9863, n = 87) is extrapolated to all requests; §7 admits per-token rates 'may shift with batch composition, prompt lengths, and system noise.'
domain assumption Lightweight LB prefix trie mirrors the serving runtime's exact KV-cache prefix state
§3.5 says the trie stores prompt-prefix-to-peer mappings rather than actual KV state; §7 concedes the estimates are approximate.
domain assumption Measured inter-LB RTT equals user-perceived forwarding latency
§3.3 uses measured RTT between load balancers; ignores intra-network queueing, jitter, and client-side last-mile latency.
domain assumption WildChat + GuideLLM synthetic mix represents real cross-region production LLM traffic
§7 states: 'We use WildChat as the primary data source ... and treat it as representative of real-world usage.' Only 10 concurrent in-flight requests are used, so a genuinely high-load regime is not demonstrated.

how reviews work

0 comments

Cite this review

Pith. "Pith review of GORGO: Online Tuning for Cross-Region Network-Aware LLM Serving." pith.science (2026). https://pith.science/paper/CRPQAYB2

@misc{pith2026260211688,
  author       = {Pith},
  title        = {Pith review of: GORGO: Online Tuning for Cross-Region Network-Aware LLM Serving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CRPQAYB2}},
  note         = {Machine review of arXiv:2602.11688}
}

read the original abstract

Increasingly, LLM inference services proxy client requests to engine replicas distributed globally. Load-balancing policies must jointly account for factors including KV-cache locality, replica load, and variable network latency when optimizing for metrics like latency and TTFT. However, existing systems only evaluate a subset of these factors in their cost model, leading to uneven concentrations of load and KV-cache across replicas. We present GORGO, a proxy architecture that holistically factors network latency, prefill cost, and queueing delay using tunable parameters. Since open-source chat datasets such as LMSYS-Chat1M and WildChat-4.8M lack long-context, high prefix-reuse data, we release a synthetic dataset, ART-Chat-2.5M, from long-context production metadata. On a tuning window from ART-Chat-2.5M, evolutionary strategies guide the GORGO policy's parameters to directly optimize p95 TTFT. During held-out evaluation windows, we fix the parameter values learned from tuning and improve p95 TTFT by 6.9-15.5% and p95 end-to-end (E2E) latency by 14.3-30.9% over baseline load-balancing policies such as simple session affinity and prefix-cache. The code and ART-Chat-2.5M dataset can be found at https://github.com/Arcadia-Research-Team/GORGO.

Figures

Figures reproduced from arXiv: 2602.11688 by the authors.

**Figure 1.** Overview of the load balancer control flow per request. but their control-plane costs and tail latency penalties grow with cluster scale and KV-cache size (Cao et al., 2025). 2.3. Geo-Distributed Inference and Cross-Region Routing Multi-region deployments are motivated by availability, latency to globally distributed users, and cost efficiency when demand shifts across regions over the day. SkyServe represents thi… view at source ↗

**Figure 2.** Median latency and throughput metrics across methods. 4.6. Benchmark and Instrumentation Each run lasts 60 seconds and processes requests from simulated users distributed globally, with 10 concurrent requests kept in flight at all times using GuideLLM. This sustained load allows us to effectively measure server behavior under pressure. We record per-request timestamps at each routing stage (load-balancer ingress/e… view at source ↗

**Figure 3.** Mean latency and throughput metrics across methods. mission delay (§3). This decomposition clarifies which signals the load balancer must track (per-token compute rates, measured RTT, and admission/queue state) and provides a common interface for integrating cache-aware routing with geo-routing. Importantly, the model is not intended to perfectly predict per-request latency; rather, it operationalizes a consistent… view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Perfetto Trace of GPU-2 During Prefix Trie Request Forwarding. 12 [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

NetKV: Network-Aware Decode Instance Selection for Disaggregated LLM Inference
cs.PF 2026-06 unverdicted novelty 5.0 of 10

NetKV is a network-aware O(|D|) greedy scheduler for decode instance selection that reduces mean TTFT by up to 21.2% versus round-robin and 17.6% versus cache+load baselines in 64-GPU fat-tree simulations.

Reference graph

Works this paper leans on

13 extracted references · 2 linked inside Pith · cited by 1 Pith paper

[2]

Belotti, J

Accessed 2026-01-28. Belotti, J. and Chang, A. Keeping 20,000 gpus healthy, December

2026
[4]

Jain, K., Parayil, A., Mallick, A., Choukse, E., Qin, X., Zhang, J., Goiri, ´I., Wang, R., Bansal, C., R ¨uhle, V ., Kulkarni, A., Kofsky, S., and Rajmohan, S

Accessed 2026-01-28. Jain, K., Parayil, A., Mallick, A., Choukse, E., Qin, X., Zhang, J., Goiri, ´I., Wang, R., Bansal, C., R ¨uhle, V ., Kulkarni, A., Kofsky, S., and Rajmohan, S. Performance aware llm load balancer for mixed workloads. InPro- ceedings of the 5th Workshop on Machine Learning and Systems (EuroMLSys ’25),

2026
[7]

Ousterhout, A., Fried, J., Behrens, J., Belay, A., and Balakr- ishnan, H

Accessed 2026-01-28. Ousterhout, A., Fried, J., Behrens, J., Belay, A., and Balakr- ishnan, H. Shenango: Achieving high cpu efficiency for latency-sensitive datacenter workloads. InProceedings of the 16th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’19),

2026
[9]

Preble: Efficient distributed prompt schedul- ing for large language model serving

Srivatsa, V ., Gao, H., Patel, M., Sivaraman, A., and Ras- togi, A. Preble: Efficient distributed prompt schedul- ing for large language model serving. arXiv preprint arXiv:2407.00023,

arXiv
[11]

Wildchat: 1m chatgpt interaction logs in the wild

Zhao, Y ., Gu, X., and Chen, D. Wildchat: 1m chatgpt interaction logs in the wild. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP),

2024
[13]

A. Benchmark Metrics 10 GORGO: Maximizing KV-Cache Reuse While Minimizing Network Latency in Cross-Region LLM Load Balancing Table 2.Least-Loaded Benchmark Metrics Metric Count Mean Median Std. Dev. Min Max P50 P90 P95 P99 TTFT (ms) 100 2087.36 568.27 4105.92 163.59 24156.60 568.27 4203.25 11541.26 18114.67 Inter-Token Latency (ms) 51820 12.08 12.46 1.34 ...
[2015]

J., Li, Z., Xing, J., Shenker, S., and Stoica, I

Xia, T., Mao, Z., Kerney, J., Jackson, E. J., Li, Z., Xing, J., Shenker, S., and Stoica, I. Skywalker: A locality-aware cross-region load balancer for LLM inference. arXiv preprint arXiv:2505.24095,

arXiv
[2019]

Si, M., Balaji, P., Chen, Y ., Chu, C.-H., Gangidi, A., Hasan, S., Iyengar, S., Johnson, D., Liu, B., Ren, R., Shah, D., Shetty, A. J., Steinbrecher, G., Wang, Y ., Wu, B., Xie, X., Yang, J., Yang, M., Yu, K., Yu, M., Zhao, C., Bland, W., Boyda, D., Gumudavelli, S., Kannan, P., Lumezanu, C., Miao, R., Qu, Z., Ramesh, V ., Samoylov, M., Seidel, J., Sundare...

arXiv

Show all 13 references

[2022]

Llm almanac: How to benchmark

Modal. Llm almanac: How to benchmark. Doc- umentation, 2026a. URL https://modal.com/ llm-almanac/how-to-benchmark . Accessed 2026-01-28. Modal. Prefix caching. Documentation, 2026b. Accessed 2026-01-28. OpenAI. Chatgpt. Large language model,

2026
[2023]

H., Cao, S., Koonce, C., Cheng, J., Gonzalez, J

Zheng, L., Yin, L., Xie, Z., Huang, J., Sun, C., Yu, C. H., Cao, S., Koonce, C., Cheng, J., Gonzalez, J. E., Sto- ica, I., and Zhang, H. Sglang: Efficient execution of structured language model programs. arXiv preprint arXiv:2312.07104,

arXiv
[2024]

Skyserve: Serving ai models across regions and clouds with spot instances

Mao, H., Yang, Z., Phanishayee, A., Mahajan, R., and Sto- ica, I. Skyserve: Serving ai models across regions and clouds with spot instances. InProceedings of the 2025 USENIX Annual Technical Conference,

2025
[2025]

Cao, Y ., Wang, S., Li, B., Liu, L., and Xu, C

URL https://modal.com/blog/ gpu-health. Cao, Y ., Wang, S., Li, B., Liu, L., and Xu, C. Dlpm: Dis- tributed load balancing for large language model infer- ence with prefix management. InProceedings of the 2025 ACM SIGMOD International Conference on Management of Data,

2025
[2026]

Anthropic

Accessed 2026- 01-28. Anthropic. Claude. Large language model,

2026

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [2]

Belotti, J

Accessed 2026-01-28. Belotti, J. and Chang, A. Keeping 20,000 gpus healthy, December

2026

[2] [4]

Jain, K., Parayil, A., Mallick, A., Choukse, E., Qin, X., Zhang, J., Goiri, ´I., Wang, R., Bansal, C., R ¨uhle, V ., Kulkarni, A., Kofsky, S., and Rajmohan, S

Accessed 2026-01-28. Jain, K., Parayil, A., Mallick, A., Choukse, E., Qin, X., Zhang, J., Goiri, ´I., Wang, R., Bansal, C., R ¨uhle, V ., Kulkarni, A., Kofsky, S., and Rajmohan, S. Performance aware llm load balancer for mixed workloads. InPro- ceedings of the 5th Workshop on Machine Learning and Systems (EuroMLSys ’25),

2026

[3] [7]

Ousterhout, A., Fried, J., Behrens, J., Belay, A., and Balakr- ishnan, H

Accessed 2026-01-28. Ousterhout, A., Fried, J., Behrens, J., Belay, A., and Balakr- ishnan, H. Shenango: Achieving high cpu efficiency for latency-sensitive datacenter workloads. InProceedings of the 16th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’19),

2026

[4] [9]

Preble: Efficient distributed prompt schedul- ing for large language model serving

Srivatsa, V ., Gao, H., Patel, M., Sivaraman, A., and Ras- togi, A. Preble: Efficient distributed prompt schedul- ing for large language model serving. arXiv preprint arXiv:2407.00023,

arXiv

[5] [11]

Wildchat: 1m chatgpt interaction logs in the wild

Zhao, Y ., Gu, X., and Chen, D. Wildchat: 1m chatgpt interaction logs in the wild. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP),

2024

[6] [13]

A. Benchmark Metrics 10 GORGO: Maximizing KV-Cache Reuse While Minimizing Network Latency in Cross-Region LLM Load Balancing Table 2.Least-Loaded Benchmark Metrics Metric Count Mean Median Std. Dev. Min Max P50 P90 P95 P99 TTFT (ms) 100 2087.36 568.27 4105.92 163.59 24156.60 568.27 4203.25 11541.26 18114.67 Inter-Token Latency (ms) 51820 12.08 12.46 1.34 ...

[7] [2015]

J., Li, Z., Xing, J., Shenker, S., and Stoica, I

Xia, T., Mao, Z., Kerney, J., Jackson, E. J., Li, Z., Xing, J., Shenker, S., and Stoica, I. Skywalker: A locality-aware cross-region load balancer for LLM inference. arXiv preprint arXiv:2505.24095,

arXiv

[8] [2019]

Si, M., Balaji, P., Chen, Y ., Chu, C.-H., Gangidi, A., Hasan, S., Iyengar, S., Johnson, D., Liu, B., Ren, R., Shah, D., Shetty, A. J., Steinbrecher, G., Wang, Y ., Wu, B., Xie, X., Yang, J., Yang, M., Yu, K., Yu, M., Zhao, C., Bland, W., Boyda, D., Gumudavelli, S., Kannan, P., Lumezanu, C., Miao, R., Qu, Z., Ramesh, V ., Samoylov, M., Seidel, J., Sundare...

arXiv