Meta-Learning for Speeding Up Large Model Inference in Decentralized Environments

Ahmad Farhan; Claudio Angione; Fielding Johnston; Harry Yang; James P. Buban; Patrick Colangelo; Yipeng Du; Yue Zhao; Yuzhe Yang; Zihao Wang

REVIEW 5 major objections 6 minor 15 references

Meta-Learning for Speeding Up Large Model Inference in Decentralized Environments

T0 review · 5 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A learned scheduler picks the best LLM acceleration method on unseen hardware with 90% accuracy and ~18% time savings.

desk verdict Plausible meta-learning for LLM inference algorithm selection, but the zero-shot table lacks the selected strategies, so the central generalization claim is not yet supported. read the letter →

arxiv 2508.09194 v1 pith:D6ZJKLI3 submitted 2025-08-08 cs.LG cs.AI

Yipeng Du , Zihao Wang , Ahmad Farhan , Claudio Angione , Harry Yang , Fielding Johnston , James P. Buban , Patrick Colangelo

show 2 more authors

Yue Zhao Yuzhe Yang

This is my paper

classification cs.LGcs.AI

keywords meta-learninginferenceaccelerationalgorithmselectionLLMservingprefixcachingcontinuousbatchingzero-shotgeneralizationdecentralized

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Inference acceleration for large language models is not one-size-fits-all: continuous batching, prefix caching, and chunked prefill each win in different regimes, and the best choice shifts with batch size, model architecture, and GPU. The paper proposes MetaInf, a meta-learning scheduler that learns from historical performance records to pick the acceleration method for a new model–hardware–workload combination. It represents each configuration by LLM-generated semantic embeddings, compresses them with truncated SVD, and trains a gradient-boosted tree regressor to predict inference time, then selects the method that maximizes predicted performance under a deployment budget. MetaInf is reported to outperform heuristic and learning-based baselines on selection accuracy (0.898 vs. 0.815), achieve a higher average acceleration ratio (1.55 vs. 1.30), and produce consistent speedups on unseen model–GPU pairs such as LLaMA-3.1-70B on H200.

What carries the argument

The load-bearing mechanism is the learned performance predictor $f: (E^{\mathrm{data}}_i, E^{\mathrm{model}}_j, E^{\mathrm{hardware}}_k) \mapsto P_{i,j,k}$, trained offline by regression on historical acceleration outcomes. Each embedding axis is an LLM-generated semantic text embedding, compressed with truncated SVD to a latent dimension of 64 to 256, so that semantically similar models and GPUs end up close in embedding space. The predictor is fit with XGBoost, chosen for robustness to heterogeneous, high-dimensional inputs. Online selection solves the constrained maximization of Eq. (2) under a budget $b$, so the selected method is the one with the highest predicted performance among thos

What would settle it

Evaluate MetaInf on a deliberately constructed set of model–GPU pairs whose text descriptions are near-duplicates of training entries but whose measured acceleration rankings are reversed (e.g., GPUs with similar names but different memory bandwidth). If selection accuracy on this set drops to the level of the heuristic baselines, then the semantic-embedding transfer does not capture the performance-relevant variation, and the central generalization claim fails.

Watch

Extended reading notes

Core claim

The central claim is that optimal inference acceleration can be chosen without online benchmarking by learning a predictor $f$ that maps embeddings of the dataset, model, and hardware to performance outcomes (Eq. 1). The framework encodes each component as a text description, embeds it with a pretrained language model, and reduces the dimensionality via truncated SVD; an XGBoost regressor is trained on historical (task, method, hardware) triples to predict runtime. At deployment, MetaInf computes the same embeddings for the new setting and selects the method with maximum predicted performance subject to a cost constraint: the selected method is the one that maximizes $f($E^{{\mathrm{data}}$}_{\m

Load-bearing premise

The framework assumes that the semantic text embeddings of a model name, GPU name, and boolean acceleration flags, after compression, preserve enough of what actually determines inference performance that the regressor's ranking transfers to hardware and models never seen in training.

Editorial extensions

If this is right

Because MetaInf selects in a zero-shot manner, deploying a new model on a new GPU requires no online benchmarking, which is the key claimed benefit for decentralized environments.
The framework is formulated over an abstract method catalog, so the same learning pipeline can incorporate additional acceleration techniques such as speculative decoding or quantization without redesign.
The budget constraint in Eq. (2) means the chosen method respects hardware cost limits, allowing the scheduler to be applied on heterogeneous nodes with different cost profiles.
Reported acceleration ratios translate directly into inference time and, hence, cost savings, supporting the paper's claim that adaptive selection makes decentralized LLM serving more economically feasible.
The speedups on an unseen GPU (H200) indicate that a single globally trained scheduler can replace device-specific tuning rules across a fleet.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural but untested extension is to use the same embedding-and-rank scheme to select among deeper optimization choices such as quantization bits, speculative-decoding drafts, or tensor-parallelism settings, since the method catalog is configurable.
Because the embeddings are built from free-text descriptions, the transfer may be sensitive to naming and phrasing conventions; a controlled test that anonymizes or rephrases GPU descriptors would reveal whether the generalization is semantic or surface-level.
The reported average acceleration ratio is a mean over candidate methods; an editorially useful complementary metric would be the worst-case ratio, since a scheduler that avoids catastrophic choices is more valuable in production.
The framework is not evaluated in a dynamically changing environment; a direct extension would be online re-selection when the hardware pool changes mid-request, leveraging the low cost of the predictor.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's the short version: MetaInf is a reasonable engineering paper about using LLM embeddings plus XGBoost to pick among inference acceleration strategies (prefix caching, chunked prefill, continuous batching) for a given model/GPU combo. The main claims are that it beats hand-tuned heuristics and standard ML baselines on selection accuracy, and that it generalizes zero-shot to unseen hardware like H200. The first claim is supported by a solid if narrow evaluation; the second is not yet supported, mainly because Table 5 never says which strategy MetaInf selected.

What is genuinely new is applying semantic embeddings to the algorithm-selection problem for LLM serving. That's a practical idea. The authors do real inference runs across T4/L4/A100, compare against the right baselines (ALORS, ARGOSMART, ISAC), and ablate prompt complexity and SVD dimensions. The 1000-trial evaluation protocol with exhaustive ground truth is a reasonable way to measure selection accuracy.

The load-bearing weakness is Table 5. It reports speedups over continuous batching for six unseen pairings, but not which acceleration strategy was chosen for each row. If MetaInf consistently picked, say, prefix caching, then the table shows only that that strategy beats continuous batching on those workloads—nothing about adaptive selection or embeddings. Also, the Time Saved column has inverted signs (Mistral-7B A100: 179.41 vs 202.90 is reported as -11.6%, which is actually a 13% improvement), so I distrust the numbers until corrected. Table 4 has no error bars or significance tests, so the 0.898 vs 0.815 accuracy gap might be noise. The experiments cover only three GPUs and four models, so broad claims of generalization are overreach. No code or data is released, which is a problem for an empirical claim like this.

On the plus side, the paper does not fake its evaluation: it actually measures runtime under stated protocols, and the ablations indicate that richer prompts and higher SVD dimensions help, which is consistent with the idea that semantic embeddings carry useful signal. The core approach is a legitimate new application of established algorithm-selection ideas.

Who is this for? Someone building a scheduling layer for decentralized LLM serving who wants a concrete baseline. As a research contribution, it's a useful data point but not a definitive one. I'd bring it to a reading group as an example of how to (and how not to) report zero-shot generalization.

Recommendation: send it to peer review if it isn't already—the topic is important and the method is sensible. But require the authors to release code and data, fix the metrics, add confidence intervals, and break out the selected strategies in the zero-shot table. I would not cite it in its current form.

Referee Report

5 major / 6 minor

Summary. The paper proposes MetaInf, a meta-learning framework that selects among inference acceleration strategies (prefix caching, chunked prefill, continuous batching, and the combination "All") for LLM inference in decentralized settings. In the offline phase, LLM-derived semantic embeddings of the model, GPU, dataset, and acceleration method are reduced by truncated SVD and used to train an XGBoost regressor on historical runtime measurements. At deployment, the regressor predicts each candidate's runtime and selects the fastest within a cost budget, without online measurement. Experiments on T4/L4/A100 report selection accuracy 0.898, F1 0.85, and an average acceleration ratio of 1.55 over learning and heuristic baselines; a zero-shot evaluation on six model-GPU pairs, including the unseen H200, is presented as evidence of generalization.

Significance. If the claimed results hold, the paper offers a practical, automated alternative to hand-picking inference acceleration strategies on heterogeneous hardware, with a clean offline/online decomposition and a meaningful zero-shot generalization probe (H200). The evaluation protocol is a genuine strength: for the main comparison, all candidate strategies are executed to obtain ground-truth labels, and selection accuracy against that ground truth is reported. The ablation of prompt style and SVD dimensionality is also valuable. However, as printed, the central quantitative claims are undermined by an inverted definition of the acceleration ratio (Table 4), sign errors in the zero-shot table (Table 5), and missing information about which strategies MetaInf actually selected in zero-shot rows. These issues must be resolved before the paper's main claims can be accepted.

major comments (5)

[§4.3, Table 4] The Average Acceleration Ratio is defined as "the ratio of inference time using the prediction model's selected solution to the average inference time across all solutions," which is lower-is-better. Yet the reported values are higher-is-better (MetaInf 1.55 vs Gradient Boosting 1.30). Either the definition should be inverted (e.g., average time / selected time, a speedup ratio) or the reported numbers contradict the definition. Since this metric is the empirical basis for the efficiency claims, the definition and the values must be reconciled, and the direction of the metric should be stated explicitly.
[§4.4, Table 5] The Time Saved column has inverted signs. For Mistral-7B A100, MetaInf 179.41s vs continuous batching 202.90s means MetaInf is 11.6% faster, i.e., +11.6% time saved, not –11.6%. For Mixtral-8x7B H200, MetaInf 39.85s vs 37.42s means MetaInf is 6.5% slower, not +6.5%. Thus the sentence "MetaInf shows consistent speedups across most settings" is not supported by the table as printed; one of the six rows is a slowdown. Please correct the signs and re-evaluate the conclusion.
[§4.2, Table 5] The zero-shot evaluation reports only MetaInf's total runtime versus continuous batching. It does not report which acceleration strategy MetaInf selected for each model-GPU pair, nor the runtimes of the other candidate strategies (prefix caching, chunked prefill, All) on those pairs. Without this information, the observed speedups could be produced by a fixed strategy (e.g., prefix caching), and the presence of adaptive selection via learned embeddings is not established. Please include, per row, the selected strategy, the runtime of every candidate strategy, and the rank of the empirically best configuration, plus a fixed-best-strategy baseline for comparison.
[§3.4, Eq. (1) and Figure 2] The notation is ambiguous: Emodel is used for the acceleration-method embedding, while the surrounding text and Eq. (1) describe the function as mapping dataset, model, and hardware embeddings to performance. In §3.3, M denotes the set of acceleration methods, not the served model. This conflation makes the exact input tuple to f unclear and could mislead readers about what is being selected. Please rename the method embedding (e.g., Emethod) and formally define the input tuple to f as (Edata, Emethod, Ehardware).
[§4.3, §4.6, Appendix A.2] No error bars, confidence intervals, or per-seed statistics are reported for Table 4 or Figure 4, although the text says results are averaged over 1000 random (model, GPU) trials. With only four models and three GPUs, the variance across trials and across random splits may be substantial. Please report standard deviations or confidence intervals for the selection accuracy, F1, and acceleration ratio, and describe the exact train/test split of model-GPU pairs used for the 1000 trials.

minor comments (6)

[Tables 2 and 3] Typo: "Chuncked Prefill" should be "Chunked Prefill."
[§3.4] "Single Value Decomposition" should be "Singular Value Decomposition" (SVD).
[§1, Table 1 reference] The text says "We further demonstrate this in 1 where Phi-2..." — "1" should be "Table 1."
[§3.1.1] The hardware list in §4.1 says experiments span T4, L4, and A100, but Table 5 adds H200. Please list H200 in the hardware enumeration and specify its memory and configuration.
[§4.2] The statement that H200 "was also absent from the embedding model's training corpus" is a factual claim that is not supported by any citation or verification. If it cannot be verified, please remove or qualify it.
[§4.6] "LLaMA-3.2B-Instruct" is ambiguous; if the intended model is LLaMA-3.2-3B-Instruct, please state the exact model name.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MetaInf is a supervised meta-predictor trained on measured performance and evaluated on held-out combinations; the claims do not reduce to their inputs by construction.

full rationale

The paper's derivation chain is a standard supervised learning pipeline, not a circular one. Historical performance entries Pi,j,k (Eq. 1) are measured empirically on actual hardware; embeddings of task, method, and hardware are derived from textual descriptions via an LLM and SVD; XGBoost is trained to regress those embeddings to the measured performance; at test time the trained predictor is applied to new inputs and the predicted best method is compared against the empirically fastest configuration (Appendix A.2). The predicted quantity is not defined in terms of any fitted parameter: the acceleration ratio and accuracy are computed from the held-out empirical evaluation after the predictor is fixed. No load-bearing step invokes a self-citation, a uniqueness theorem, or an ansatz imported from prior work by the same authors; the method is not a renaming of a known result. The legitimate concerns raised by the skeptic—Table 5 does not report which strategy MetaInf selected, and Table 4 may not enforce a strict split over model–GPU pairs—are experimental reporting and leakage questions, not definitional circularity. Under the hard rule that circularity must be exhibited by quoting an equation or self-citation reduction, none is present here.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper's contribution rests on two empirical bets: transferable embeddings and the quality of its runtime oracle. The only hand-fitted numbers are the SVD dimension and prompt template, neither of which is tied to the target result by construction. No new entities are introduced.

free parameters (3)

SVD dimensionality k = 64, 128, 256; default 256
Chosen by ablation in Section 4.6; higher k improves accuracy up to 256. This is a modeling hyperparameter fitted on the validation tasks.
Prompt style for embeddings = not specified; Basic/Rich/CoT in ablation
Section 4.6 ablates three prompt templates; the default used in Table 4 is not fixed in the text, so the reported main results depend on an unspecified prompt choice.
XGBoost hyperparameters = not reported
The meta-predictor in Section 3.4 is XGBoost; learning rate, tree depth, regularization, and number of estimators are not given, making the Table 4 results hard to reproduce.

assumptions (4)

domain assumption Historical runtime measurements on sampled tasks are representative of, and predictive for, new deployment conditions.
The meta-learning objective in Eq. (1) assumes transfer from a database of historical tasks to unseen tasks; this is the core inductive bias of the framework.
domain assumption LLM embeddings capture task-model-hardware compatibility relevant to inference speed.
Used in Section 4.2; if embeddings are semantically about names rather than performance-relevant properties, zero-shot ranking fails.
domain assumption The custom unified inference engine implements continuous batching, prefix caching, and chunked prefill correctly and comparably to production systems.
Motivating measurements in Section 3.1 and all training targets in the performance matrix P depend on these implementations.
standard math Truncated SVD preserves sufficient signal from high-dimensional LLM embeddings.
Standard linear-algebra dimensionality reduction, assumed in Section 4.2; low-rank approximation is a mathematical fact but its adequacy for this task is empirical.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Meta-Learning for Speeding Up Large Model Inference in Decentralized Environments." pith.science (2026). https://pith.science/paper/D6ZJKLI3

@misc{pith2026250809194,
  author       = {Pith},
  title        = {Pith review of: Meta-Learning for Speeding Up Large Model Inference in Decentralized Environments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D6ZJKLI3}},
  note         = {Machine review of arXiv:2508.09194}
}

read the original abstract

The deployment of large-scale models, such as large language models (LLMs), incurs substantial costs due to their computational demands. To mitigate these costs and address challenges related to scalability and data security, there is a growing shift towards decentralized systems for model deployment, where choosing efficient inference acceleration schemes become crucial to manage computational resources effectively and enhance system responsiveness. In this work, we address the challenge of selecting optimal acceleration methods in decentralized systems by introducing a meta-learning-based framework. This framework automates the selection process by learning from historical performance data of various acceleration techniques across different tasks. Unlike traditional methods that rely on random selection or expert intuition, our approach systematically identifies the best acceleration strategies based on the specific characteristics of each task. We demonstrate that our meta-learning framework not only streamlines the decision-making process but also consistently outperforms conventional methods in terms of efficiency and performance. Our results highlight the potential of inference acceleration in decentralized AI systems, offering a path towards more democratic and economically feasible artificial intelligence solutions.

Figures

Figures reproduced from arXiv: 2508.09194 by the authors.

**Figure 1.** Performance comparison of acceleration strategies across varying batch sizes. We compare the performance of different fast inference techniques on the LLaMA 3.1 8B model. The combined strategy outperforms all individual techniques for small batch sizes, while prefix caching performs best for larger batches. This highlights the need for adaptive selection mechanisms tailored to workload scale. settings. While effecti… view at source ↗

**Figure 2.** MetaInf overview (§3.3). The top illustrates offline meta-training (§3.4), where a predictor f (shown as ) learns to map embeddings of datasets and models to performance outcomes P. The bottom shows online selection (§3.5), where f is used to predict performance for new tasks and hardware settings. parallelism—without costly empirical evaluations. This selection must account for hardwarespecific constraints and co… view at source ↗

**Figure 3.** Performance of our MetaInf and baseline methods on select the best-performing acceleration combination. This bar plot shows the average rank of each method when choosing the true best configuration across all tasks. Lower values indicate more reliable and consistent performance; MetaInf (ours) ranks first with the best performance. encodings to assess the impact of embedding choice, our default approach employs … view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Performance comparison of MetaInf and baseline methods across different embedding schemes. We evaluate model selectors under four embedding schemes—Onehot, Basic, Rich, and Chain-of-Thought (CoT)—using configuration selection accuracy (left) and average speedup improv…

**Figure 5.** Figure 5: Efficiency–effectiveness trade-off of prediction methods. We compare the Efficiency-effectiveness between the accuracy of the tested prediction methods and the inference time of each prediction to measure the performance of our proposed MetaInf compared to other method…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

15 extracted references · 7 canonical work pages

[2]

Adaptive Orchestration for Large-Scale Inference on Heterogeneous Accelerator Systems Balancing Cost, Performance, and Resilience

URL https://arxiv.org/abs/2503.20074. Alexander Borzunov, Max Ryabinin, Artem Chumachenko, Dmitry Baranchuk, Tim Dettmers, Younes Belkada, Pavel Samygin, and Colin A Raffel. Distributed inference and fine-tuning of large language models over the internet. Advances in Neural Information Processing Systems, 36,

work page Pith review arXiv
[5]

Jerome H Friedman

URL https://arxiv.org/abs/2503.11023. Jerome H Friedman. Greedy function approximation: A gradient boosting machine. Annals of Statistics, 29(5):1189–1232,

arXiv
[6]

Scaling laws for autore- gressive generative modeling

Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autore- gressive generative modeling. arXiv preprint arXiv:2010.14701,

arXiv 2010
[10]

Hierarchical text-conditional image generation with clip latents

Aditya Ramesh et al. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125,

arXiv
[11]

ZeRO-Offload: Democratizing billion-scale model training

Shaoduo Ren, Xinyuan Zeng, Shuxiong Liu, Fan Yang, Olatunji Ruwase, Amar Phanishayee Rao, and Yuxiong Zhang. ZeRO-Offload: Democratizing billion-scale model training. arXiv preprint arXiv:2101.06840,

arXiv
[12]

Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro

URL https://arxiv.org/abs/2503.08223. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053,

arXiv 1909
[14]

Mingjin Zhang, Xiaoming Shen, Jiannong Cao, Zeyang Cui, and Shan Jiang

URL https://arxiv.org/abs/2406.01394. Mingjin Zhang, Xiaoming Shen, Jiannong Cao, Zeyang Cui, and Shan Jiang. Edgeshard: Efficient llm inference via collaborative edge computing. IEEE Internet of Things Journal ,

arXiv
[15]

,n} do 3: Extract data embedding Edata i = ψ(Di) 4: for j ∈ {1,

12 Published as a conference paper at COLM 2025 A Appendix A.1 Pseudo-code for Meta-train and Online Model Selection Algorithm 1 Offline Meta-Learner Training for Inference Acceleration Input: Meta-train set Dtrain, model set M, hardware set H Output: Meta-learner f for acceleration method selection 1: Train and evaluate M across H on Dtrain to get perfor...

work page 2025

Show all 15 references

[2001]

Language models are few-shot learners

10 Published as a conference paper at COLM 2025 Tom Brown et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901,

2025
[2017]

Algorithm selection for combinatorial search problems: A survey

Pascal Kerschke, Holger H Hoos, Kevin Leyton-Brown, and Heike Trautmann. Algorithm selection for combinatorial search problems: A survey. arXiv preprint arXiv:1811.04563,

arXiv
[2019]

Geeps: Scalable deep learning on distributed gpus with a gpu-specialized parameter server

Hao Xu, Cheuk Yiu Ho, Ahmed M Abdelmoniem, Aritra Dutta, El Houcine Bergou, Kon- stantinos Karatsenidis, Marco Canini, and Panos Kalnis. Geeps: Scalable deep learning on distributed gpus with a gpu-specialized parameter server. In Proceedings of the Fourteenth EuroSys Conferen...

2019
[2020]

Medusa: Simple llm inference acceleration framework with multiple decoding heads

Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple llm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10774,

arXiv
[2022]

Eagle: Speculative sampling requires rethinking feature uncertainty

Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprint arXiv:2401.15077,

arXiv
[2023]

Zero: Memory optimization towards training a trillion parameter models

11 Published as a conference paper at COLM 2025 Samyam Rajbhandari, Jordan Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimization towards training a trillion parameter models. International Conference for High Performance Computing, Networking, Storage and Analysis ,

2025
[2025]

Michele Belotti, Nikola Boˇzi´c, Guy Pujolle, and Stefano Secci

URL https://arxiv.org/abs/ 2504.10693. Michele Belotti, Nikola Boˇzi´c, Guy Pujolle, and Stefano Secci. Vademecum: A decentralized peer-to-peer network for cloud and edge computing. Future Generation Computer Systems , 94:868–878,

arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [2]

Adaptive Orchestration for Large-Scale Inference on Heterogeneous Accelerator Systems Balancing Cost, Performance, and Resilience

URL https://arxiv.org/abs/2503.20074. Alexander Borzunov, Max Ryabinin, Artem Chumachenko, Dmitry Baranchuk, Tim Dettmers, Younes Belkada, Pavel Samygin, and Colin A Raffel. Distributed inference and fine-tuning of large language models over the internet. Advances in Neural Information Processing Systems, 36,

work page Pith review arXiv

[2] [5]

Jerome H Friedman

URL https://arxiv.org/abs/2503.11023. Jerome H Friedman. Greedy function approximation: A gradient boosting machine. Annals of Statistics, 29(5):1189–1232,

arXiv

[3] [6]

Scaling laws for autore- gressive generative modeling

Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autore- gressive generative modeling. arXiv preprint arXiv:2010.14701,

arXiv 2010

[4] [10]

Hierarchical text-conditional image generation with clip latents

Aditya Ramesh et al. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125,

arXiv

[5] [11]

ZeRO-Offload: Democratizing billion-scale model training

Shaoduo Ren, Xinyuan Zeng, Shuxiong Liu, Fan Yang, Olatunji Ruwase, Amar Phanishayee Rao, and Yuxiong Zhang. ZeRO-Offload: Democratizing billion-scale model training. arXiv preprint arXiv:2101.06840,

arXiv

[6] [12]

Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro

URL https://arxiv.org/abs/2503.08223. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053,

arXiv 1909

[7] [14]

Mingjin Zhang, Xiaoming Shen, Jiannong Cao, Zeyang Cui, and Shan Jiang

URL https://arxiv.org/abs/2406.01394. Mingjin Zhang, Xiaoming Shen, Jiannong Cao, Zeyang Cui, and Shan Jiang. Edgeshard: Efficient llm inference via collaborative edge computing. IEEE Internet of Things Journal ,

arXiv

[8] [15]

,n} do 3: Extract data embedding Edata i = ψ(Di) 4: for j ∈ {1,

12 Published as a conference paper at COLM 2025 A Appendix A.1 Pseudo-code for Meta-train and Online Model Selection Algorithm 1 Offline Meta-Learner Training for Inference Acceleration Input: Meta-train set Dtrain, model set M, hardware set H Output: Meta-learner f for acceleration method selection 1: Train and evaluate M across H on Dtrain to get perfor...

work page 2025