Pith. sign in

REVIEW 2 major objections 6 minor 35 references

Scalable Machine Learning Training Infrastructure for Online Ads Recommendation and Auction Scoring Modeling at Google

T0 review · 2 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that production ad-model training on TPUs can be made 116 percent faster and 18 percent cheaper by sharing input generation, overlapping embedding and dense computation, and cutting wasted accelerator time.

desk verdict A credible, well-scoped industry systems paper with one load-bearing gap: the pipelining speedup rests on an unquantified claim that stale embedding gradients don't hurt model quality. read the letter →

arxiv 2501.10546 v1 pith:UIHZW6WO submitted 2025-01-17 cs.DC cs.AIcs.LG

George Kurian , Somayeh Sardashti , Ryan Sims , Felix Berger , Gary Holt , Yang Li , Jeremiah Willcock , Kaiyuan Wang
show 3 more authors
Herve Quiroz Abdulrahman Salem Julian Grady
This is my paper · ORCID
classification cs.DCcs.AIcs.LG
keywords TPUtraininginfrastructureembeddingpartitioningsharedinputgenerationrecommendationmodelsauctionscoringcostreductionpreemptionhandlingcontinuous
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

At production scale, the bottleneck in training ad recommendation and auction-scoring models is not the matrix math on accelerators but the data plumbing around it. This paper argues that three software techniques remove those bottlenecks: memoizing shared feature transformations so many models reuse the same preprocessed inputs, partitioning and overlapping embedding lookups with dense computation, and checkpointing cleanly through preemptions and errors. On five representative production models that cover more than half of the authors' ad-training workloads, the combined system trains 116 percent faster and costs 18 percent less. A careful reader should care because these are the practical differences between idle or wasted accelerator time and useful learning.

What carries the argument

The load-bearing mechanisms are the input-generation memoization and the embedding pipeline overlap. SIG reuses materialized outputs of shared connected components of each model's transformation graph, achieving cache hit rates above 95 percent and each memoized block used by about 22 models. The embedding machinery is a constraint-optimized hybrid of table, column, and row partitioning enriched by feedback-directed profiling, combined with pipelining that runs SparseCore embedding work for one step while TensorCore dense work runs for the previous step, trading exact gradient timing for overlap. For caught-up training, fetch/update coalescing collapses all parameter-server traffic into one serialized RPC pair per training step, with table stacking and reference-counted splits to avoid memory copies.

What would settle it

Take one of the five representative models and train it twice on identical data and step counts, once with serialized SparseCore/TensorCore execution and once with pipelining, then compare held-out prediction metrics at the same checkpoint schedule; a statistically significant drop under pipelining would refute the quality-neutrality premise that the speedup claims lean on.

Watch

Extended reading notes

Core claim

The central claim is that a production training system can get nearly all of the available speedup from orchestration rather than new model math. The paper describes an infrastructure where a shared input generation service precomputes connected components of feature-transformation graphs and reuses them across training pipelines, while per-model input readers scale horizontally to keep accelerators fed. Large embedding tables are spread across TPU memory with a hybrid mix of row, column, and table partitioning chosen by a constraint solver using runtime profiling; the SparseCore lookups for step N+1 are overlapped with TensorCore dense computation for step N, accepting one-step-stale gradients; and caught-up training on parameter servers is accelerated by coalescing all embeddings into one RPC pair per step. Around this core, preemption notices and training holds prevent checkpoint loss and keep expensive accelerators from idling. The authors' evaluation attributes a 58 to 180 percent throughput gain (geometric mean 116 percent) to the embedding optimizations and a 4.3 to 7.5 fold reduction in input-reader cost, totaling an 18 percent training-cost reduction, to shared input generation.

Load-bearing premise

The load-bearing premise is that overlapping SparseCore and TensorCore work by one training step does not degrade final model quality; the paper asserts this for internal recommendation and ranking models but provides no quantitative comparison, and if staleness hurts quality, the reported 116 percent speedup is not a real efficiency gain.

Editorial extensions

If this is right

  • On the five representative models, the embedding optimizations alone improve training throughput by 58 to 180 percent, with a geometric mean of 116 percent, and models that are already TensorCore-bound gain less.
  • Shared input generation lowers input-reader cost by 4.3 to 7.5 times, and with TPU cost dominating, this reduces total training-pipeline cost by 12 to 27 percent (geometric mean 18 percent).
  • During caught-up training, fetch/update coalescing improves speed by 6 to 10 percent, making small-slice continuous training viable without losing parameter-server efficiency.
  • With preemption notices, 61 percent of preempted training epochs still commit their checkpoints, so progress made before interruption is not discarded.
  • Training holds release accelerator resources for queued jobs: models on hold demand 2.49 times the active chip footprint, which would otherwise strand expensive hardware.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not report training-quality numbers for the one-step-stale gradients, but its logic suggests a testable extension: measure whether staleness stays quality-neutral for longer pipelines or different model families, since that would show whether the overlap trick transfers.
  • The shared-input-generation approach implies a cost model where memoization pays off when transformation components are reused widely; the reported 95 percent hit rate and 22-model reuse suggest similar shared preprocessing services could be tried for search or feed ranking with the same hit-rate metrics.
  • Feedback-directed partitioning depends on cross-model feature statistics accumulating over time, so a testable prediction is that a newly introduced model reaches its peak speed only after enough batches have populated the profiling database.
  • The reported 18 percent cost reduction excludes cases where LIG's CPU hunger made TPUs idle, so in resource-constrained settings the real savings from SIG could be materially larger.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. This paper describes Google's production training infrastructure for Ads recommendation and auction-scoring models on TPUs. It presents a shared input generation service (SIG) that memoizes and amortizes feature transformations across models, a horizontally scaled input reader service, hybrid table/column/row partitioning with feedback-directed profiling for large embedding tables on TPU SparseCores, software pipelining between TensorCore and SparseCore, RPC coalescing for CPU-resident embeddings, and preemption-notice and training-hold mechanisms for shared datacenters. The evaluation on five representative production models reports a 116% geometric-mean throughput improvement from the embedding optimizations (range 58%-180%) and an 18% geometric-mean total training-cost reduction from SIG (range 12%-27%), together with supporting measurements of input-reader CPU ratios, SIG hit rates, and TPU-chip demand.

Significance. If the technical claims hold, this is a valuable industrial case study: it documents concrete, non-obvious systems techniques (hybrid partitioning with runtime profiling, single-pair RPC coalescing for parameter servers, and a memoized shared input-generation service) at production scale. The paper is candid about several limitations of SIG and about the non-ideal SIG/LIG comparison, which is a credit to the authors. The architecture descriptions are detailed enough to inform practitioners building similar infrastructure. However, the headline quantitative claims are self-reported, carry no error bars or per-model numerical tables, and depend critically on an unquantified assumption that one-step-stale embedding gradients do not affect final model quality. The paper therefore currently provides a plausible and well-argued systems narrative, but the central efficiency numbers are not yet fully substantiated.

major comments (2)
  1. [V.C, Figure 5, VII.D] The 116% geometric-mean throughput gain is conditional on the quality-neutrality of stale embedding gradients. Pipelining changes training semantics: the SparseCore begins step N+1 while the TensorCore executes step N, making embedding gradients stale by one step. The paper asserts twice (V.C: 'was found to not impact model quality in internal recommendation and ranking models'; VII.D: 'there was no noticeable quality impact in all models studied') that this is benign, but provides no offline or live quality metrics, no description of the comparison protocol, no confidence intervals, and no identification of which of the five representative models were included in those experiments. Since pipelining is enabled for all models in the VII.D evaluation, the measured throughput improvement is an efficiency gain only if final model quality is unchanged; otherwise fewer training steps do not imply reaching the same quality. Please report the actual quality metrics (e.g., offline AUC/log loss, or live metric deltas) with and without pipelining for the five representative models, or at minimum state precisely which models were covered by the staleness experiments.
  2. [VII.B, Figure 6] The 18% cost-reduction claim is confounded by the admitted resource scarcity in the LIG baseline. The text states: 'It wasn't practical to obtain sufficient CPU and RAM resources for all LIG experiments. When external CPU/RAM resources are low, we found that TPU utilization suffered... When TPU utilization is low, the model training cost with LIG is greatly increased.' This mechanism inflates the measured LIG cost and therefore inflates SIG's apparent benefit, yet the text concludes 'the benefits with SIG were in reality much higher,' which appears to be in the opposite direction of the described bias. Please clarify the direction of the bias and provide a controlled comparison where LIG is given sufficient resources, or a sensitivity analysis showing how the 12%-27% range changes under different resource-availability assumptions. Per-model cost components (TPU vs. input reader) and error bars for Figure 6 should also be reported.
minor comments (6)
  1. [VII.D, Figure 7] Figure 7 lacks visible axis labels and error bars, and the text does not specify the number of measurements per bar. Adding a table with absolute per-model step times for baseline, pipelining, hybrid partitioning, and FDP would make the 58%-180% range and the per-model decomposition auditable.
  2. [VII.B, Figure 6] The y-axis of Figure 6 is described only in the caption as 'normalized to LIG'; please state explicitly that the normalization is to LIG total cost and, if possible, include confidence intervals or per-model value tables.
  3. [VII.G] The 61% preemption-commit rate is reported without defining the denominator (all preempted epochs, or only those with preemption notices) and without a baseline for comparison. Please clarify the measurement window and the definition.
  4. [IX, reference [6]] Reference [6] is cited in the text for 'data preprocessing services built on TensorFlow's tf.data,' but the reference is Audibert et al., 'A case for disaggregation of ML data processing.' Please confirm this is the intended citation or replace it with the tf.data reference.
  5. [V.A, Eq. (1)] The parenthetical in the load-imbalance example, '2 (= 4×0.6 / 0.6+0.3+0.2+0.1)', is ambiguous; it should be written as 4×0.6 / (0.6+0.3+0.2+0.1).
  6. [V.A, VII.A] Minor wording: 'N-P hard' should be 'NP-hard' in Section V.A, and 'comprised of' in Section VII.A should be 'composed of'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the headline numbers are direct production measurements, not outputs of a fitted model or self-referential derivation.

full rationale

This is a systems paper reporting measured production improvements, not a mathematical derivation. The headline 116% throughput gain and 18% training-cost reduction are direct measurements on five representative models, with the cost reduction explicitly computed as a TCO comparison (Section VII.B) and the speedup measured against a defined serialized baseline (Section VII.D). No parameter is fitted and then renamed as a prediction; the feedback-directed partitioning uses profiling statistics to configure partitioning, but the reported speedups are measured after applying that configuration, not predicted from those statistics. The paper does rely on self-citations for background hardware and model architecture (e.g., TPU v4 [15], ads model references), but these are not load-bearing in the sense of making a result true by definition; the TPU v4 hardware characteristics are externally described and the evaluation is self-contained against internal baselines. The stale-gradient quality-neutrality assertion in Sections V.C and VII.D is an unsupported empirical claim and a correctness/verifiability risk, but it is not circular: it is an assertion of an experimental finding, not a claim derived from an input that already contains the conclusion. Similarly, the admitted difficulty of running complete LIG baselines (Section VII.B) weakens the strength of the SIG comparison but does not make any derivation circular. No equation reduces to its own input, and no fitted quantity is exhibited as a prediction.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces system components (SIG, input readers, training holds) that are software artifacts implemented at Google, not unexplained physical entities, so the invented_entities list is empty. The central quantitative claims depend on the four domain/accounting assumptions listed above; none of them is a fitted parameter in the usual sense, so free_parameters is empty. The 116% and 18% figures are summary statistics of measurements, not constants tuned to fit the conclusions.

assumptions (4)
  • domain assumption TPU v2/v3/v4 hardware characteristics (SparseCore behavior, HBM/ICI bandwidth, host CPU limits) are as cited from Google's prior papers.
    Section II.B and Section VII.A assume the TPU hardware behaves as described in TPU v2/v3 [23] and TPU v4 [15] publications; if those descriptions are inaccurate, the partitioning and pipelining analysis breaks down.
  • domain assumption The five representative models account for >50% of production workloads and their measured improvements generalize.
    Section VII.A states the models were picked for the study; the paper gives no evidence that the speedup figures transfer to the remaining workloads.
  • ad hoc to paper The TCO model (CapEx normalized over machine lifetime plus CPU, RAM, tray, power costs) correctly captures the cost comparisons in Figure 6.
    Section VII.A defines the cost metric used for the 18% reduction claim; different cost accounting (e.g., including storage, network, engineering time) could change the number.
  • ad hoc to paper One-step-stale embedding gradients do not degrade final model quality.
    Section V.C asserts this from internal experiments without presenting data; it supports the decision to enable pipelining across all five models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scalable Machine Learning Training Infrastructure for Online Ads Recommendation and Auction Scoring Modeling at Google." pith.science (2026). https://pith.science/paper/UIHZW6WO

@misc{pith2026250110546,
  author       = {Pith},
  title        = {Pith review of: Scalable Machine Learning Training Infrastructure for Online Ads Recommendation and Auction Scoring Modeling at Google},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UIHZW6WO}},
  note         = {Machine review of arXiv:2501.10546}
}
read the original abstract

Large-scale Ads recommendation and auction scoring models at Google scale demand immense computational resources. While specialized hardware like TPUs have improved linear algebra computations, bottlenecks persist in large-scale systems. This paper proposes solutions for three critical challenges that must be addressed for efficient end-to-end execution in a widely used production infrastructure: (1) Input Generation and Ingestion Pipeline: Efficiently transforming raw features (e.g., "search query") into numerical inputs and streaming them to TPUs; (2) Large Embedding Tables: Optimizing conversion of sparse features into dense floating-point vectors for neural network consumption; (3) Interruptions and Error Handling: Minimizing resource wastage in large-scale shared datacenters. To tackle these challenges, we propose a shared input generation technique to reduce computational load of input generation by amortizing costs across many models. Furthermore, we propose partitioning, pipelining, and RPC (Remote Procedure Call) coalescing software techniques to optimize embedding operations. To maintain efficiency at scale, we describe novel preemption notice and training hold mechanisms that minimize resource wastage, and ensure prompt error resolution. These techniques have demonstrated significant improvement in Google production, achieving a 116% performance boost and an 18% reduction in training costs across representative models.

Figures

Figures reproduced from arXiv: 2501.10546 by the authors.

Figure 1
Figure 1. High level components of the training service, with corresponding [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Example connected component from an Ads recommendation model’s [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. System design and architecture of the shared input generation (SIG) [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Embedding Table Partitioning strategies for TPU. There are three model parallelism strategies used: (a) table partitioning, (b) column partitioning, and [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: (a) Serialized and (b) Pipelined execution of TensorCore with Spar [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Cost of the training service. The cost of input readers is much higher [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Impact of embedding optimizations on the performance of recom [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 24 canonical work pages

  1. [1]

    https://www.ibm.com/docs/en/pessl/5.3.0?topic=distributions- distribution-techniques

    work page

  2. [2]

    https://en.wikichip.org/wiki/amd/cores/rome

    work page

  3. [3]

    Understanding training efficiency of deep learning recommendation models at scale,

    B. Acun, M. Murphy, X. Wang, J. Nie, C.-J. Wu, and K. Hazelwood, “Understanding training efficiency of deep learning recommendation models at scale,” in 2021 IEEE International Symposium on High- Performance Computer Architecture (HPCA) . IEEE, 2021, pp. 802– 814

    work page 2021

  4. [4]

    On the factory floor: Ml engineering for industrial-scale ads recommendation models,

    R. Anil, S. Gadanho, D. Huang, N. Jacob, Z. Li, D. Lin, T. Phillips, C. Pop, K. Regan, G. I. Shamir et al. , “On the factory floor: Ml engineering for industrial-scale ads recommendation models,” arXiv preprint arXiv:2209.05310, 2022

    arXiv 2022

  5. [5]

    Memory efficient adaptive optimization,

    R. Anil, V . Gupta, T. Koren, and Y . Singer, “Memory efficient adaptive optimization,” Advances in Neural Information Processing Systems , vol. 32, 2019

    work page 2019

  6. [6]

    tf.data service: A Case for Disaggregating ML Input Data Processing

    A. Audibert, Y . Chen, D. Graur, A. Klimovic, J. Simsa, and C. A. Thekkath, “A case for disaggregation of ml data processing,” arXiv preprint arXiv:2210.14826, 2022

    work page Pith review arXiv 2022

  7. [7]

    cudnn: Efficient primitives for deep learning,

    S. Chetlur, C. Woolley, P. Vandermersch, J. Cohen, J. Tran, B. Catanzaro, and E. Shelhamer, “cudnn: Efficient primitives for deep learning,” arXiv preprint arXiv:1410.0759, 2014

    arXiv 2014

  8. [8]

    Large scale distributed deep networks,

    J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, M. Ran- zato, A. Senior, P. Tucker, K. Yang et al., “Large scale distributed deep networks,” Advances in neural information processing systems , vol. 25, 2012

    2012

Show all 35 references
  1. [9]

    Materialization and reuse optimizations for production data science pipelines,

    B. Derakhshan, A. Rezaei Mahdiraji, Z. Kaoudi, T. Rabl, and V . Markl, “Materialization and reuse optimizations for production data science pipelines,” in Proceedings of the 2022 International Conference on Management of Data , 2022, pp. 1962–1976

    2022

  2. [10]

    Mtia: First generation silicon targeting meta’s recommendation systems,

    A. Firoozshahian, J. Coburn, R. Levenstein, R. Nattoji, A. Kamath, O. Wu, G. Grewal, H. Aepala, B. Jakka, B. Dreyer et al. , “Mtia: First generation silicon targeting meta’s recommendation systems,” in Proceedings of the 50th Annual International Symposium on Computer Architec...

    2023

  3. [11]

    Cachew: Machine learning input data processing as a service,

    D. Graur, D. Aymon, D. Kluser, T. Albrici, C. A. Thekkath, and A. Klimovic, “Cachew: Machine learning input data processing as a service,” in 2022 USENIX Annual Technical Conference (USENIX ATC 22), 2022, pp. 689–706

    2022

  4. [12]

    Nectar: automatic management of data and computation in datacen- ters,

    P. K. Gunda, L. Ravindranath, C. A. Thekkath, Y . Yu, and L. Zhuang, “Nectar: automatic management of data and computation in datacen- ters,” in 9th USENIX Symposium on Operating Systems Design and Implementation (OSDI 10) , 2010

    2010

  5. [13]

    Practical lessons from predicting clicks on ads at facebook,

    X. He, J. Pan, O. Jin, T. Xu, B. Liu, T. Xu, Y . Shi, A. Atallah, R. Herbrich, S. Bowers et al., “Practical lessons from predicting clicks on ads at facebook,” in Proceedings of the eighth international workshop on data mining for online advertising , 2014, pp. 1–9

    2014

  6. [14]

    Xdl: an industrial deep learning framework for high-dimensional sparse data,

    B. Jiang, C. Deng, H. Yi, Z. Hu, G. Zhou, Y . Zheng, S. Huang, X. Guo, D. Wang, Y . Songet al., “Xdl: an industrial deep learning framework for high-dimensional sparse data,” in Proceedings of the 1st International Workshop on Deep Learning Practice for High-Dimensional Sparse...

    2019

  7. [15]

    Tpu v4: An optically reconfigurable supercomputer for machine learning with hardware sup- port for embeddings,

    N. Jouppi, G. Kurian, S. Li, P. Ma, R. Nagarajan, L. Nai, N. Patil, S. Subramanian, A. Swing, B. Towles et al. , “Tpu v4: An optically reconfigurable supercomputer for machine learning with hardware sup- port for embeddings,” in Proceedings of the 50th Annual International Sym...

    2023

  8. [16]

    Persia: An open, hybrid system scaling deep learning-based recommenders up to 100 trillion parameters,

    X. Lian, B. Yuan, X. Zhu, Y . Wang, Y . He, H. Wu, L. Sun, H. Lyu, C. Liu, X. Dong et al. , “Persia: An open, hybrid system scaling deep learning-based recommenders up to 100 trillion parameters,” in Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data...

    2022

  9. [17]

    Monolith: real time recommendation system with collisionless embedding table,

    Z. Liu, L. Zou, X. Zou, C. Wang, B. Zhang, D. Tang, B. Zhu, Y . Zhu, P. Wu, K. Wang et al. , “Monolith: real time recommendation system with collisionless embedding table,” arXiv preprint arXiv:2209.07663 , 2022

    2022 arXiv

  10. [18]

    Ad click prediction: a view from the trenches,

    H. B. McMahan, G. Holt, D. Sculley, M. Young, D. Ebner, J. Grady, L. Nie, T. Phillips, E. Davydov, D. Golovin et al., “Ad click prediction: a view from the trenches,” in Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining , 2013, ...

    2013

  11. [19]

    Efficient estimation of word representations in vector space,

    T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word representations in vector space,” arXiv preprint arXiv:1301.3781 , 2013

    2013 arXiv

  12. [20]

    Module: tf.math — tensorflow,

    “Module: tf.math — tensorflow,” https://www.tensorflow.org/api docs/ python/tf/math/segment sum, Accessed 2023-10-26

    2023

  13. [21]

    Software-hardware co-design for fast and scalable training of deep learning recommendation models,

    D. Mudigere, Y . Hao, J. Huang, Z. Jia, A. Tulloch, S. Sridharan, X. Liu, M. Ozdal, J. Nie, J. Park et al. , “Software-hardware co-design for fast and scalable training of deep learning recommendation models,” in Proceedings of the 49th Annual International Symposium on Comput...

    2022

  14. [22]

    Deep learning recommendation model for personalization and recommenda- tion systems,

    M. Naumov, D. Mudigere, H.-J. M. Shi, J. Huang, N. Sundaraman, J. Park, X. Wang, U. Gupta, C.-J. Wu, A. G. Azzolini et al. , “Deep learning recommendation model for personalization and recommenda- tion systems,” arXiv preprint arXiv:1906.00091 , 2019

    1906 arXiv

  15. [23]

    The design process for google’s training chips: Tpuv2 and tpuv3,

    T. Norrie, N. Patil, D. Yoon, G. Kurian, S. Li, J. Laudon, C. Young, N. Jouppi, and D. Patterson, “The design process for google’s training chips: Tpuv2 and tpuv3,” IEEE Micro, vol. PP, pp. 1–1, 02 2021

    2021

  16. [24]

    The design process for google’s training chips: Tpuv2 and tpuv3,

    T. Norrie, N. Patil, D. H. Yoon, G. Kurian, S. Li, J. Laudon, C. Young, N. Jouppi, and D. Patterson, “The design process for google’s training chips: Tpuv2 and tpuv3,” IEEE Micro, vol. 41, no. 2, pp. 56–63, 2021

    2021

  17. [25]

    Uplift: parallelization strategies for feature transformations in machine learning workloads,

    A. Phani, L. Erlbacher, and M. Boehm, “Uplift: parallelization strategies for feature transformations in machine learning workloads,” Proceedings of the VLDB Endowment , vol. 15, no. 11, pp. 2929–2938, 2022

    2022

  18. [26]

    Recshard: statistical feature-based memory optimization for industry-scale neural recommendation,

    G. Sethi, B. Acun, N. Agarwal, C. Kozyrakis, C. Trippel, and C.- J. Wu, “Recshard: statistical feature-based memory optimization for industry-scale neural recommendation,” in Proceedings of the 27th ACM International Conference on Architectural Support for Programming Language...

    2022

  19. [27]

    Flexshard: Flexible sharding for industry-scale sequence recommenda- tion models,

    G. Sethi, P. Bhattacharya, D. Choudhary, C.-J. Wu, and C. Kozyrakis, “Flexshard: Flexible sharding for industry-scale sequence recommenda- tion models,” arXiv preprint arXiv:2301.02959 , 2023

    2023 arXiv

  20. [28]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,

    N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” arXiv preprint arXiv:1701.06538 , 2017

    2017 arXiv

  21. [29]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017

    2017

  22. [30]

    Large-scale cluster management at google with borg,

    A. Verma, L. Pedrosa, M. Korupolu, D. Oppenheimer, E. Tune, and J. Wilkes, “Large-scale cluster management at google with borg,” in Proceedings of the tenth european conference on computer systems , 2015, pp. 1–17

    2015

  23. [31]

    El-rec: Efficient large-scale recommendation model training via tensor- train embedding table,

    Z. Wang, Y . Wang, B. Feng, D. Mudigere, B. Muthiah, and Y . Ding, “El-rec: Efficient large-scale recommendation model training via tensor- train embedding table,” in SC22: International Conference for High Performance Computing, Networking, Storage and Analysis . IEEE, 2022, pp. 1–14

    2022

  24. [32]

    Agile and accurate ctr prediction model training for massive-scale online ad- vertising systems,

    Z. Xu, D. Li, W. Zhao, X. Shen, T. Huang, X. Li, and P. Li, “Agile and accurate ctr prediction model training for massive-scale online ad- vertising systems,” in Proceedings of the 2021 international conference on management of data , 2021, pp. 2404–2409

    2021

  25. [33]

    Large batch optimiza- tion for deep learning: Training bert in 76 minutes,

    Y . You, J. Li, S. Reddi, J. Hseu, S. Kumar, S. Bhojanapalli, X. Song, J. Demmel, K. Keutzer, and C.-J. Hsieh, “Large batch optimiza- tion for deep learning: Training bert in 76 minutes,” arXiv preprint arXiv:1904.00962, 2019

    1904 arXiv

  26. [34]

    Wukong: Towards a scaling law for large-scale recommendation,

    B. Zhang, L. Luo, Y . Chen, J. Nie, X. Liu, D. Guo, Y . Zhao, S. Li, Y . Hao, Y . Yaoet al., “Wukong: Towards a scaling law for large-scale recommendation,” arXiv preprint arXiv:2403.02545 , 2024

    2024 arXiv

  27. [35]

    Understanding data storage and ingestion for large-scale deep recommendation model training: Industrial product,

    M. Zhao, N. Agarwal, A. Basant, B. Gedik, S. Pan, M. Ozdal, R. Ko- muravelli, J. Pan, T. Bao, H. Lu et al., “Understanding data storage and ingestion for large-scale deep recommendation model training: Industrial product,” in Proceedings of the 49th annual international sympos...

    2022

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.