Pith. sign in

REVIEW 4 major objections 4 minor 28 references

PyroDash claims that a small language model can learn to decide during generation whether to hand off its partial reasoning trace to a frozen large model, and that this token-level decision improves the accuracy–cost trade-off compared with

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 10:07 UTC pith:O72FLWIU

load-bearing objection The method is genuinely new, but the key experiment lacks a no-handoff GRPO control, so the central claim is not yet established. the 4 major comments →

arxiv 2607.20327 v1 pith:O72FLWIU submitted 2026-07-22 cs.CL

PyroDash: Cost-Efficient Token-Level Small-Large Language Model Collaborative Inference

Niqi Lyu , Pengtao Shi , Wei Qiu , Jianlin Zhong , Sicong Xia , Jianyao Ma , Yicheng Ding This is my paper
classification cs.CL
keywords SLM-LLM collaborationtoken-level routingcontrol tokenGRPOinference costcollaborative inferenceoffloading policymathematical reasoning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

PyroDash claims that a small language model can learn to decide during generation whether to hand off its partial reasoning trace to a frozen large model, and that this token-level decision — made by emitting a special control token — produces a better accuracy–cost trade-off than either model alone. On five math benchmarks, the accuracy-oriented policy reaches 64.04% average accuracy, 6.36 points above the large-model-only baseline, while cutting estimated cost by 20.4%. The cost-oriented policy keeps average accuracy at 54.55% while reducing the estimated bill from $49.36 to $1.78, a 96.4% reduction, with only 1.90% of decoded tokens coming from the large model. Training uses three stages ending in a cost-aware reinforcement-learning step, and needs no separate router, no retraining of the large model, and no access to its internal probabilities. If the results hold, token-level self-routing could make large-model assistance affordable for high-volume reasoning workloads.

Core claim

PyroDash places the offloading decision inside the small model's own generation stream: the small model emits a dedicated control token when it needs help, and a Collaborate Engine passes query plus partial trace to the frozen large model for a single continuation. A reward of answer accuracy minus a normalized cost penalty, trained with Group Relative Policy Optimization (GRPO), yields a family of policies: at low penalty average accuracy beats the large-model-only baseline (64.04% vs 57.68%) while cost drops; at high penalty the large model's token share falls to 1.90% and the estimated bill drops from $49.36 to $1.78. The three-stage pipeline conditions the token's use on query difficulty

What carries the argument

The key mechanism is the control token τ_off, a new vocabulary token whose embedding is initialized as a noisy mean of boundary-token embeddings (period, newline, end-of-sequence). During decoding, the small model can emit τ_off at a capability boundary; a Collaborate Engine detects it, packages query and partial trace, and sends them to the frozen large model for a single continuation. Training proceeds in three stages: control-token embedding learning, offloading-oriented supervised fine-tuning, and GRPO alignment with reward accuracy − λ·(collaborative cost / LLM-only cost). Sweeping λ yields operating points from near-full large-model reliance (λ=0.05) to near-full small-model autonomy (

Load-bearing premise

The reported accuracy and cost improvements assume the training corpora are disjoint from the five evaluation benchmarks; the paper reports no overlap or contamination check, so any leaked test problems would inflate both the accuracy gains and the cost savings.

What would settle it

Run the trained policies on a new math benchmark released after the training data were collected, and also search the training sets for exact or near-duplicate problems from the five test suites; if a meaningful number of test problems appear in training, the 64.04% accuracy point and the 96.4% cost reduction would not be reproducible on fresh data.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Token-level, single-handoff collaboration is feasible: the small model's output stream itself carries the routing signal, so no separate router or repeated calls are needed.
  • Cost can be cut dramatically: at λ=0.6 the estimated bill falls from $49.36 to $1.78 (96.4%) while average accuracy stays at 54.55%, within 3.13 points of the large-model-only baseline.
  • Accuracy can improve while cost falls: at λ=0.05 average accuracy exceeds the large-model-only baseline by 6.36 points and cost drops by 20.4%.
  • The λ parameter provides a tunable accuracy–cost frontier: the large model's token share falls from 95.34% to 1.90% as λ increases from 0.05 to 0.6, with average accuracy declining by about 9 points.
  • The large model stays frozen and only a standard generation API is needed, making the approach compatible with proprietary models and opaque endpoints.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same token-level trigger could generalize beyond math to code, tool use, or multimodal subproblems where a small model can identify a specific step that needs a specialist; the paper does not test those settings.
  • The dollar figures come from listed token prices, not actual provider bills, so real-world savings could be smaller once fixed fees, caching, and per-request overhead are counted.
  • The paper does not show that handoffs occur precisely at the reasoning step where the small model fails; analyzing trigger positions would reveal whether the policy is genuinely capability-aware or merely cost-driven.
  • No training/test overlap check is reported, so the headline gains should be re-verified on a fresh benchmark released after the training data were collected.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes PyroDash, a token-level collaborative inference framework in which a small language model (SLM) learns to emit a control token τ_off during decoding; a Collaborate Engine then performs a single handoff to a frozen LLM, which completes the reasoning from the SLM's partial trace. The SLM is trained in three stages: control-token embedding learning, offloading-oriented SFT, and GRPO with a reward that balances answer accuracy against a normalized inference-cost penalty controlled by λ. Experiments on GSM8K, Minerva, OlympiadBench, AIME25, and AIME24 report 64.04% average accuracy at λ=0.05 (6.36 pp above LLM-only, 20.4% cost reduction) and 54.55% average accuracy at λ=0.6 with 0.012 LLM calls/example and total cost reduced from $49.36 to $1.78. The method requires no separate router, no LLM retraining, and no access to LLM logits.

Significance. If the empirical claims hold, PyroDash would be a practical and lightweight way to combine an inexpensive SLM with a frozen API-only LLM, with a tunable accuracy-cost trade-off. The paper's strengths include a clear formalization of the single-handoff protocol (Section 3.2, Algorithm 1), a transparent cost accounting model (Equations 4–7 and 11), and a three-stage training pipeline that is well specified. The reported cost decomposition in Table 4 is also a useful contribution. However, the central attribution of the accuracy-cost frontier to the token-level handoff is currently under-supported: the cost-oriented operating points show very few LLM calls, and no ablation isolates the handoff's contribution from GRPO's effect on the SLM's own policy. Missing contamination checks, absent token-level baselines, and lack of variance estimates further weaken the headline numbers.

major comments (4)
  1. [§4.2, §4.4, Tables 1–3] At λ=0.6, Avg. LLM Calls = 0.012 per example (Table 1). Since each handoff can change the outcome of at most one example, the maximum possible aggregate contribution of the handoff to average accuracy is 1.2 percentage points. Yet PyroDash λ=0.6 reaches 54.55% average accuracy versus 46.25% for SLM+SFT, an 8.30 pp gain. Even at λ=0.1, with 0.058 calls/example (max 5.8 pp), the gain over SFT is 9.04 pp. Therefore most of the accuracy improvement in the cost-oriented operating points comes from GRPO updating the SLM's own generation policy, not from token-level handoffs. The Q2 ablation in §4.2 only compares SFT with PyroDash at λ=0.05, where handoffs are frequent (0.975 calls/example). A no-handoff GRPO control — e.g., Stage 3 rollouts that never invoke the LLM, or disabling τ_off at inference — is required to establish that the reported accuracy-cost frontier is due to the learned handof
  2. [§3.3.1, §4.1.3] The training corpora EasyHard-24k (Stages 1–2) and DAPO-Math (Stage 3) are not checked for overlap with the five evaluation benchmarks: GSM8K, Minerva, OlympiadBench, AIME25, and AIME24. EasyHard-24k is constructed from collected examples with LLM reconstruction, but the paper reports no exact-match, n-gram, or embedding-level contamination analysis. If test problems or near-duplicates appear in training, the +6.36 pp gain at λ=0.05 and the 54.55% accuracy at λ=0.6 would be inflated. This is load-bearing for the central empirical claim; please report overlap statistics and provide results on a contamination-cleaned subset.
  3. [§2 vs §4.1.2] The related work discusses CITER, Co-LLM, and RelayLLM as token-level collaborative-inference baselines that are closest to PyroDash, but none of these appears in the experiments. The only evaluated baselines are request-level RouteLLM and step-level GlimpRouter, both of which allocate >75% of decoded tokens to the LLM. Without the closest token-level baselines, the claim that PyroDash advances token-level collaboration is not directly tested. Additionally, all accuracy numbers are point estimates without confidence intervals or multiple seeds; AIME results are avg@32, so the 6.36 pp headline difference should be accompanied by variance estimates.
  4. [Eqs. (5), (7), and (11)] The training reward uses Cost_act in Eq. (5) with a unified per-token SLM price C_s and counts every SLM output token also as LLM prefill, while evaluation cost in Eq. (11) uses four separate prices (SLM input/output, LLM input/output). The paper does not specify how C_s relates to p_slm_out and p_llm_in, so the objective optimized during GRPO may not align with the reported dollar costs. This weakens the interpretation of the λ sweep as a clean accuracy-cost control. Please either align the two cost models or justify the compact form and show that it preserves the ordering of policies under the evaluation prices.
minor comments (4)
  1. [Eq. (10)] The LLM Token Ratio excludes prefill tokens. Since prefill is a major cost driver in Table 4, consider also reporting an LLM input-token fraction or the full cost decomposition in Eq. (11) for the λ=0.05 operating point, where the 20.4% cost reduction is not broken down.
  2. [§3.3.2] The description of Stage 1 merging is ambiguous: 'merge only the trained parameter rows corresponding to τ_off' could mean the new token embedding row is kept while other modified rows are discarded, but the text also says the embedding layer and output head are fully fine-tuned. Please clarify exactly which parameters are retained after the transient LoRA adapters are removed.
  3. [§4.1.3, Appendix A] The qualitative examples are illustrative only; please state whether they were selected after seeing outcomes, to avoid concerns about cherry-picking. Also, the model names Qwen3.5-4B and GLM-5.2-FP8 appear to be newly introduced; ensure stable version identifiers are provided for reproducibility.
  4. [§6] The limitations section acknowledges that offloading decisions are not analyzed and that the normalized cost penalty is not compared to a direct token-cost penalty. These are appropriate caveats, but the missing no-handoff GRPO ablation (Major Comment 1) should also be listed as a limitation, since it directly affects the interpretation of the main results.

Circularity Check

0 steps flagged

No significant circularity: the accuracy-cost results are empirical outcomes of the stated GRPO reward, not equations that reduce to their inputs; only minor in-house references keep the score above zero.

full rationale

The paper's derivation chain runs from the objective in Eq. (1) through the three-stage pipeline (Eqs. (2)-(9)) to the measured results in Tables 1-4. The reward terms are not fitting targets in disguise: R_acc is computed by extracting \boxed{} answers and applying equivalence checks, and R_eff is computed from token counts and listed prices (Eqs. (4)-(7), (11)). Neither term is algebraically identical to the reported accuracy or cost figures. The lambda values (0.05, 0.1, 0.6) are selected via a validation-set sweep, which is standard hyperparameter selection, not fitting the reported evaluation results into the objective. GRPO optimizes Eq. (7) over sampled collaborative rollouts, so the reported accuracy and cost points are empirical outcomes rather than constructed identities. The strongest concerns in the paper - no no-handoff GRPO control at lambda=0.6, and no reported contamination check between EasyHard-24k/DAPO-Math training data and the five benchmarks - are experimental-design and data-provenance issues, not circularity: no equation is shown to reduce to another by construction, and no fitted constant is renamed as a prediction. The paper's in-house citations (EasyHard-24k [PyroMind Dynamics, 2026b], PyroMind platform [PyroMind Dynamics, 2026a]) support training data and infrastructure, but the paper describes the curation and training setup itself in Sections 3.3.1 and 4.1.3, so the central claim does not rest on an unverified self-citation. Score 2 reflects only minor in-house references and validation-set selection, below the threshold for substantive circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 4 invented entities

The paper's central claims rest on a pricing-model assumption, a data-disjointness assumption, a baseline-fairness assumption, and a training-data-validity assumption. The only public artifact is the EasyHard-24k dataset link; control token, CE, and platform are not independently verifiable from the preprint. The free parameters are mostly standard training knobs plus the lambda sweep, which is a tunable operating point rather than a hidden fitted constant.

free parameters (4)
  • lambda (efficiency weight) = 0.05, 0.1, 0.6
    Coefficient in Equation 7; swept on validation set and used to select the reported accuracy-cost operating points.
  • sigma (embedding noise) = 0.1
    Standard deviation of Gaussian noise in Equation 2 for tau_off embedding initialization; hand-set.
  • LoRA ranks, alpha, learning rates, epochs = r=8/16, alpha=16/32, lr=1e-5/5e-6, epochs 4/1
    Training hyperparameters in Section 4.1.3; they affect results but are not fit to test data.
  • GRPO group size G = 8
    Number of rollouts per group in Equation 8; affects advantage normalization.
axioms (4)
  • domain assumption Token prices are linear and additive (Eq. 11), and SLM output tokens before handoff are charged as LLM prefill (Eq. 5).
    The reward and all dollar-savings numbers follow this pricing model; Section 6 says costs are based on listed prices rather than actual provider bills.
  • domain assumption The training corpora (EasyHard-24k and DAPO-Math) are disjoint from the five evaluation benchmarks.
    No leakage/overlap analysis is reported; if test problems appear in training data, accuracy and cost gains would be inflated.
  • domain assumption The LLM-only baseline is a fair comparison reference with comparable prompting to the collaborative path.
    The baseline prompt is not shown, so part of the +6.36pp gain at lambda=0.05 could come from the staged prefix rather than from the routing policy itself.
  • ad hoc to paper Easy/hard labels and LLM-reconstructed targets in EasyHard-24k are valid supervision for when offloading is useful.
    The dataset is created by the authors for this method; its quality and label validity are not independently benchmarked.
invented entities (4)
  • tau_off (control token) no independent evidence
    purpose: Lets the SLM signal a handoff inside its own token stream; the CE detects it and stops SLM decoding.
    Observable only through paper-reported traces; no trained model or checkpoint is released to independently verify the trigger semantics.
  • Collaborate Engine (CE) no independent evidence
    purpose: Detects tau_off, packs the partial context C_s, and invokes the frozen LLM once through a completion API.
    Only Algorithm 1 is described; no source code or binary is released.
  • EasyHard-24k dataset independent evidence
    purpose: Training data with easy/hard subsets and inserted tau_off labels for Stages 1-2.
    A public Hugging Face link (pyromind/easyhard-24k) is provided, so independent auditors can download and inspect it.
  • PyroMind platform no independent evidence
    purpose: Proprietary training and serving infrastructure used for the reported experiments.
    Mentioned as the provider; no independent verification of its claimed billing or throughput properties is available.

pith-pipeline@v1.3.0-alltime-deepseek · 19862 in / 14466 out tokens · 116614 ms · 2026-08-01T10:07:36.070978+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) provide strong reasoning capabilities but are expensive to serve at scale, whereas small language models (SLMs) are cheaper but less reliable on difficult problems. We introduce PyroDash, a cost-aware framework for token-level SLM-LLM collaborative inference. During generation, the SLM decides whether to request assistance by emitting a control token. A Collaborate Engine then sends the query and partial reasoning trace to a frozen LLM for completion through a single handoff. The policy is internalized in the SLM, requiring neither a separate router, LLM retraining, nor access to LLM logits. PyroDash trains the SLM in three stages: control-token embedding learning, offloading-oriented supervised fine-tuning, and cost-aware alignment with Group Relative Policy Optimization. Its reward balances answer accuracy against inference cost normalized by LLM-only inference. Across five mathematical reasoning benchmarks, PyroDash supports different accuracy-cost operating points. With $\lambda=0.05$, it achieves 64.04 percent average accuracy, 6.36 percentage points above the LLM-only baseline, while reducing cost by 20.4 percent. With $\lambda=0.6$, it achieves 54.55 percent accuracy with a 1.90 percent LLM token ratio and 0.012 LLM calls per example, reducing total cost from USD 49.36 to USD 1.78. These results show that learned token-level handoffs can reduce LLM use while preserving strong reasoning performance.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

28 extracted references · 19 linked inside Pith

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Josh Achiam, Steven Adler, Sandhini Agarwal, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Pith/arXiv arXiv

  2. [3]

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, et al

    URLhttps://proceedings.mlr.press/v235/cai24b.html. Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, et al. Accelerating large language model decoding with speculative sampling.arXiv preprint arXiv:2302.01318,

    Pith/arXiv arXiv

  3. [4]

    Zhixiong Chen, Bingjie Zhu, Jiangzhou Wang, Hyundong Shin, Arumugam Nallanathan, and Dusit Niyato

    arXiv:2305.05176. Zhixiong Chen, Bingjie Zhu, Jiangzhou Wang, Hyundong Shin, Arumugam Nallanathan, and Dusit Niyato. Network edge inference for large language models: Principles, techniques, and opportunities.ACM Com- puting Surveys, 58(12),

    Pith/arXiv arXiv

  4. [6]

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh

    arXiv:2404.14618. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training quantiza- tion for generative pre-trained transformers. InInternational Conference on Learning Representations,

    Pith/arXiv arXiv

  5. [8]

    ChaoqunHe, Renjie Luo, YuzhuoBai, Shengding Hu, ZhenThai, JunhaoShen, JinyiHu, XuHan, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun

    arXiv:2306.08543. ChaoqunHe, Renjie Luo, YuzhuoBai, Shengding Hu, ZhenThai, JunhaoShen, JinyiHu, XuHan, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. OlympiadBench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. InProceedings of the 62nd Annual Meeting of the Association...

    Pith/arXiv arXiv

  6. [9]

    doi: 10.18653/v1/2024.acl-long.211

    Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.211. URLhttps://aclanthology.org/2024.acl-long.211/. ChengsongHuang,TongZheng,LanglinHuang,JinyuanLi,HaolinLiu,andJiaxinHuang. RelayLLM:Efficient reasoning via collaborative decoding.arXiv preprint arXiv:2601.05167,

    arXiv 2024

  7. [10]

    AIME 2024 dataset.https://huggingface.co/datasets/HuggingFaceH4/aime_2024,

    Hugging Face H4. AIME 2024 dataset.https://huggingface.co/datasets/HuggingFaceH4/aime_2024,

    2024

  8. [11]

    arXiv:2309.06180

    doi: 10.1145/3600006.3613165. arXiv:2309.06180. Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decod- ing. InProceedings of the 40th International Conference on Machine Learning, volume 202 ofProceedings of Machine Learning Research, pages 19274–19286. PMLR,

    arXiv

  9. [12]

    arXiv:2211.17192. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language mod- els. InAdvances in Neural Information Processing Systems, volume ...

    Pith/arXiv arXiv

  10. [15]

    AIME 2025 dataset.https://huggingface.co/datasets/yentinglin/aime_2025,

    Yen-Ting Lin. AIME 2025 dataset.https://huggingface.co/datasets/yentinglin/aime_2025,

    2025

  11. [16]

    Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E

    doi: 10.1145/3620666.3651335. Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M. Waleed Kadous, and Ion Stoica. RouteLLM: Learning to route LLMs with preference data. InInternational Con- ference on Learning Representations,

    arXiv

  12. [17]

    PyroMind Dynamics

    arXiv:2406.18665. PyroMind Dynamics. PyroMind console.https://pyromind.ai/, 2026a. PyroMind Dynamics. EasyHard-24K v0.02.https://huggingface.co/datasets/pyromind/easyhard-24k, 2026b. Qwen Team. Qwen3.5: Towards native multimodal agents.https://qwen.ai/blog?id=qwen3.5, February

    Pith/arXiv arXiv

  13. [18]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, et al

    URLhttps://proceedings.iclr.cc/paper_files/paper/2025/hash/ 13f972adf12bdf886583d48cd528002f-Abstract-Conference.html. Zhihong Shao, Peiyi Wang, Qihao Zhu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Pith/arXiv arXiv 2025

  14. [19]

    arXiv:2403.03870

    doi: 10.18653/v1/2024.acl-long.701. arXiv:2403.03870. Hugo Touvron, Louis Martin, Kevin Stone, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288,

    Pith/arXiv arXiv 2024

  15. [21]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H

    arXiv:2502.18482. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large lan- guage models. InAdvances in Neural Information Processing Systems, volume 35, pages 24824–24837,

    Pith/arXiv arXiv

  16. [22]

    Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhifang Sui

    URLhttps://proceedings.neurips.cc/paper_files/paper/2022/hash/ 9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html. Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhifang Sui. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding. InFindings of the Associ...

    2022

  17. [23]

    arXiv:2401.07851

    doi: 10.18653/v1/2024.findings-acl.456. arXiv:2401.07851. An Yang, Baosong Yang, Binyuan Hui, et al. Qwen2 technical report.arXiv preprint arXiv:2407.10671,

    Pith/arXiv arXiv 2024

  18. [24]

    DAPO: An open-source LLM rein- forcement learning system at scale.arXiv preprint arXiv:2503.14476,

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, et al. DAPO: An open-source LLM rein- forcement learning system at scale.arXiv preprint arXiv:2503.14476,

    Pith/arXiv arXiv

  19. [25]

    GlimpRouter: Efficientcollaborativeinferencebyglimpsingonetokenofthoughts

    WenhaoZeng,XutengZhang,YulingShi,ChaoHu,YutingChen,BeijunShen,andXiaodongGu. GlimpRouter: Efficientcollaborativeinferencebyglimpsingonetokenofthoughts. InFindingsoftheAssociationforCompu- tational Linguistics: ACL 2026, pages 17850–17864, San Diego, California, United States,

    2026

  20. [26]

    doi: 10.18653/v1/2026.findings-acl.885

    Association for Computational Linguistics. doi: 10.18653/v1/2026.findings-acl.885. arXiv:2601.05110. Wenhao Zheng, Yixiao Chen, Weitong Zhang, Souvik Kundu, Yun Li, Zhengzhong Liu, Eric P. Xing, Hongyi Wang, and Huaxiu Yao. CITER: Collaborative inference for efficient large language model decoding with token-level routing. InConference on Language Modeling,

    Pith/arXiv arXiv 2026

  21. [27]

    Work in Progress 16 PyroDash: Cost-Efficient Token-Level Small-Large Model Collaborative Inference A

    arXiv:2502.01976. Work in Progress 16 PyroDash: Cost-Efficient Token-Level Small-Large Model Collaborative Inference A. Qualitative Case Studies of the PyroDash Inference Flow ThisappendixcomplementstheaggregateresultsinSection4withfiveillustrativecasesdrawnfromtheactive PyroDash case set. Four cases show how an SLM prefix is preserved across a one-shot h...

    Pith/arXiv arXiv 2024

  22. [28]

    Takeaway.The trace separates model roles cleanly: the SLM supplies the governing equation, while the LLM completes the BCC-specific substitution, unit conversion, and rounding

    2 ≈181.3g mol −1. Takeaway.The trace separates model roles cleanly: the SLM supplies the governing equation, while the LLM completes the BCC-specific substitution, unit conversion, and rounding. A.3. Handoff after Reducing a Word Problem to an Inequality GSM8K #12: distinguishing break-even from positive profit. User prompt Carlos pays $90 to plant a lemo...

    2024

  23. [2020]

    Xinyuan Wang, Yanchi Liu, Wei Cheng, Xujiang Zhao, Zhengzhang Chen, Wenchao Yu, Yanjie Fu, and Haifeng Chen

    Apache-2.0 licensed software. Xinyuan Wang, Yanchi Liu, Wei Cheng, Xujiang Zhao, Zhengzhang Chen, Wenchao Yu, Yanjie Fu, and Haifeng Chen. MixLLM: Dynamic routing in mixed large language models. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume ...

    2025

  24. [2022]

    Senyao Li, Haozhao Wang, Wenchao Xu, Rui Zhang, Song Guo, Jingling Yuan, Xian Zhong, Tianwei Zhang, and Ruixuan Li

    URLhttps://proceedings.neurips.cc/paper_files/paper/2022/file/ 18abbeef8cfe9203fdf9053c9c4fe191-Paper-Conference.pdf. Senyao Li, Haozhao Wang, Wenchao Xu, Rui Zhang, Song Guo, Jingling Yuan, Xian Zhong, Tianwei Zhang, and Ruixuan Li. Collaborative inference and learning between edge SLMs and cloud LLMs: A survey of algorithms, execution, and open challeng...

    Pith/arXiv arXiv 2022

  25. [2023]

    Gemma Team

    arXiv:2210.17323. Gemma Team. Gemma 4 technical report.arXiv preprint arXiv:2607.02770,

    Pith/arXiv arXiv

  26. [2024]

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D

    arXiv:2306.13649. Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. Medusa: Simple LLM inference acceleration framework with multiple decoding heads. InProceedings of the 41st International Conference on Machine Learning, volume 235 ofProceedings of Machine Learning Research, pages 5209–5235,

    Pith/arXiv arXiv

  27. [2025]

    LLMbandit: Cost-efficientLLMgenerationviapreference-conditioneddynamicrouting.arXivpreprint arXiv:2502.02743,

    YangLi. LLMbandit: Cost-efficientLLMgenerationviapreference-conditioneddynamicrouting.arXivpreprint arXiv:2502.02743,

    Pith/arXiv arXiv

  28. [2026]

    arXiv:2604.22906

    doi: 10.1145/3809166. arXiv:2604.22906. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168,

    Pith/arXiv arXiv