pith. sign in

arxiv: 2604.13034 · v1 · submitted 2026-04-14 · 💻 cs.DC · cs.DB

DySkew: Dynamic Data Redistribution for Skew-Resilient Snowpark UDF Execution

Chenwei Xie , Urjeet Shrestha , Corbin McElhanney , Lukas Lorimer , Gopal V , Zihao Ye , Yi Pan , Nic Crouch
show 3 more authors
Elliott Brossard Florian Funke Yuxiong He
This is my paper

Pith reviewed 2026-05-10 13:51 UTC · model grok-4.3

classification 💻 cs.DC cs.DB
keywords data skewSnowparkUDF executiondynamic redistributionstate machinesrow size modeldistributed computingSnowflake
0
0 comments X

The pith

DySkew dynamically redistributes data at runtime using per-link state machines to counter skew in Snowpark UDF executions.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces DySkew as a skew-resilient strategy for Snowpark user-defined functions that replaces static round-robin partitioning with runtime adaptation. It builds on Snowflake's generalized skew handling by adding per-link state machines to monitor data flows and trigger redistributions, plus an eager strategy and Row Size Model tailored to large rows and arbitrary user code costs. This setup targets fine-grained per-row mitigation and low-overhead balancing so that uneven data or expensive Python logic no longer creates straggler tasks. A sympathetic reader would care because skew remains a dominant source of unpredictable latency in elastic data warehouses, and a lightweight dynamic fix would raise throughput without requiring changes to user code.

Core claim

DySkew is a novel data-skew-aware execution strategy for Snowpark UDFs built on an adaptive data distribution mechanism that uses per-link state machines for dynamic runtime monitoring and redistribution, augmented by an eager redistribution strategy and a Row Size Model to manage overhead for extremely large rows, thereby replacing static round-robin methods and delivering measurable gains in execution time and resource utilization for large-scale workloads with non-uniform user logic.

What carries the argument

Adaptive data distribution mechanism that relies on per-link state machines to detect skew in real time and decide on cost-aware redistributions, extended for Snowpark by the Row Size Model and eager triggering.

If this is right

  • Replaces static round-robin partitioning with dynamic adjustments that respond to observed data and computation imbalance during execution.
  • Enables fine-grained per-row mitigation for user-defined logic whose cost is unknown in advance.
  • Supports runtime adaptation to changing skew patterns without requiring offline analysis or code changes.
  • Keeps added overhead bounded even when individual rows are very large through explicit size modeling.
  • Improves overall resource utilization in elastic compute environments by reducing straggler impact.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The per-link state machine pattern could transfer to other UDF platforms that already support elastic scaling, provided they expose comparable flow monitoring hooks.
  • Pairing the Row Size Model with simple cost estimators for common Python operations might further reduce reactive moves.
  • The approach leaves open whether very frequent small redistributions in mildly skewed workloads ever become net-negative.

Load-bearing premise

The per-link state machines and Row Size Model can detect skew and carry out redistribution with low enough overhead to produce net gains for arbitrary user code without creating new bottlenecks.

What would settle it

Run identical large-scale Snowpark UDF jobs on highly skewed input with DySkew enabled versus the prior static distribution and check whether total wall-clock time decreases after subtracting the measured redistribution cost.

Figures

Figures reproduced from arXiv: 2604.13034 by Chenwei Xie, Corbin McElhanney, Elliott Brossard, Florian Funke, Gopal V, Lukas Lorimer, Nic Crouch, Urjeet Shrestha, Yi Pan, Yuxiong He, Zihao Ye.

Figure 1
Figure 1. Figure 1: Round Robin Per Row Redistribution This solution shows improvements both on industry standard benchmark like TPCX-BB [9] and real customer workloads, however, we also notice a few limitations with this approach: Suboptimal distribution: The redistribution is static by the nature of round robin approach, the rows will be evenly distributed to each worker [10], However, an even distribution of rows is not op… view at source ↗
Figure 2
Figure 2. Figure 2: State machine for Generalized Skew Handling [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
read the original abstract

Snowflake revolutionized data warehousing with an elastic architecture that decouples compute and storage, enabling scalable solutions for diverse data analytics needs. Building on this foundation, Snowflake has advanced its AI Data Cloud vision by introducing Snowpark, a managed turnkey solution that supports data engineering and AI/ML workloads using Python and other programming languages. While Snowpark's User-Defined Function (UDF) execution model offers high throughput, it is highly vulnerable to performance degradation from data skew, where uneven data partitioning causes straggler tasks and unpredictable latency. The non-uniform computational cost of arbitrary user code further exacerbates this classic challenge. This paper presents DySkew, a novel, data-skew-aware execution strategy for Snowpark UDFs. Built upon Snowflake's new generalized skew handling solution, an adaptive data distribution mechanism utilizing per-link state machines. DySkew addresses the unique challenges of user-defined logic with goals of fine-grained per-row mitigation, dynamic runtime adaptation, and low-overhead, cost-aware redistribution. Specifically, for Snowpark, we introduce crucial optimizations, including an eager redistribution strategy and a Row Size Model to dynamically manage overhead for extremely large rows. This dynamic approach replaces the limitations of the previous static round-robin method. We detail the architecture of this framework and showcase its effectiveness through performance evaluations and real-world case studies, demonstrating significant improvements in the execution time and resource utilization for large-scale Snowpark UDF workloads.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper introduces DySkew, a dynamic skew-handling framework for Snowpark UDF execution on Snowflake. It replaces static round-robin partitioning with per-link state machines that perform fine-grained, runtime-adaptive data redistribution, augmented by an eager redistribution strategy and a Row Size Model to control overhead for large rows. The central claim is that this yields significant improvements in execution time and resource utilization for skewed, arbitrary user-defined workloads.

Significance. If the low-overhead claims are substantiated, DySkew would address a practical bottleneck in elastic cloud data platforms that support Python and other UDFs, improving predictability for data-engineering and ML pipelines. The architectural focus on per-link adaptation and cost-aware decisions aligns with ongoing needs in distributed execution engines.

major comments (2)
  1. [Abstract] Abstract: the assertion of 'low-overhead, cost-aware redistribution' and 'eager redistribution strategy' is load-bearing for the net-gain claim, yet the text supplies no bound on per-link state-machine communication cost, no description of how the Row Size Model is trained or updated at runtime, and no indication of behavior when UDF cost is data-dependent rather than row-size-dependent.
  2. [Performance evaluations] Performance evaluations (referenced in the abstract): no quantitative results, error bars, ablation data, or overhead measurements are supplied, so it is impossible to verify whether the dynamic mechanisms deliver net gains over static round-robin for light UDFs or mild skew.
minor comments (1)
  1. [Abstract] The abstract would be clearer if it briefly stated the experimental platform, workload characteristics, and primary metrics (e.g., latency reduction, CPU utilization) used to demonstrate effectiveness.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive and detailed feedback on our manuscript. We appreciate the identification of areas where additional clarity and evidence are needed to strengthen the claims. We address each major comment below and commit to revisions that will incorporate the requested details and data.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the assertion of 'low-overhead, cost-aware redistribution' and 'eager redistribution strategy' is load-bearing for the net-gain claim, yet the text supplies no bound on per-link state-machine communication cost, no description of how the Row Size Model is trained or updated at runtime, and no indication of behavior when UDF cost is data-dependent rather than row-size-dependent.

    Authors: We agree that the abstract is too concise and does not provide sufficient supporting detail for these claims. In the revised version we will expand the abstract to include: (1) a brief statement that per-link state-machine communication is bounded by O(1) messages per batch due to the finite-state design; (2) that the Row Size Model is trained offline on representative workloads and updated periodically via lightweight sampling; and (3) that when UDF cost deviates significantly from row-size dependence, DySkew conservatively applies eager redistribution. These points are already elaborated in Sections 3.2, 4.1 and 5.3 of the full manuscript; we will also add a short limitations paragraph in the abstract. revision: yes

  2. Referee: [Performance evaluations] Performance evaluations (referenced in the abstract): no quantitative results, error bars, ablation data, or overhead measurements are supplied, so it is impossible to verify whether the dynamic mechanisms deliver net gains over static round-robin for light UDFs or mild skew.

    Authors: We acknowledge that the current manuscript text does not embed the concrete quantitative results, error bars, ablation studies or overhead numbers referenced in the abstract. In the revision we will add a new subsection (or expand Section 6) that presents: execution-time speedups across skew levels, resource-utilization metrics, error bars from repeated runs, ablation results isolating the state-machine and Row Size Model contributions, and direct overhead measurements for light UDFs and mild skew. These data will allow readers to verify net gains relative to static round-robin. revision: yes

Circularity Check

0 steps flagged

No circularity: architectural description without derivations or fitted parameters

full rationale

The paper presents DySkew as an architectural framework for dynamic skew handling in Snowpark UDFs, built on per-link state machines and a Row Size Model. No equations, mathematical derivations, predictions from fitted inputs, or self-referential definitions appear in the provided text. Claims of low-overhead redistribution and eager strategies are descriptive rather than derived from prior results within the paper. Self-citation of Snowflake's generalized solution is mentioned but does not bear load on any derivation chain, as none exists. The work is self-contained as a systems design with performance evaluations, not a closed-form or fitted-result claim.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 2 invented entities

The approach rests on the assumption that runtime state machines can track skew at per-link granularity and that a lightweight row-size model suffices to control overhead; these are domain assumptions rather than proven properties.

axioms (2)
  • domain assumption Data skew from non-uniform user code is a dominant source of stragglers in Snowpark UDF execution.
    Stated directly in the abstract as the core problem being solved.
  • ad hoc to paper Per-link state machines can adaptively redistribute data with fine-grained per-row control and low cost.
    Central mechanism introduced by the paper; no independent evidence supplied in abstract.
invented entities (2)
  • DySkew framework no independent evidence
    purpose: Dynamic skew-resilient UDF execution engine
    New system name and architecture presented in the paper.
  • Row Size Model no independent evidence
    purpose: Estimate overhead for extremely large rows to decide redistribution cost
    Optimization component introduced to handle edge cases in UDF workloads.

pith-pipeline@v0.9.0 · 5594 in / 1412 out tokens · 35830 ms · 2026-05-10T13:51:33.480542+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

16 extracted references · 16 canonical work pages

  1. [1]

    Snowpark: Performant, Secure, User -Friendly Data Engineering and AI/ML Next To Your Data,

    B. Baker et al., "Snowpark: Performant, Secure, User -Friendly Data Engineering and AI/ML Next To Your Data," in 2025 IEEE 45th International Conference on Distributed Computing Systems Workshops (ICDCSW), Glasgow, United Kingdom, 2025, pp. 213 -218, doi: 10.1109/ICDCSW63273.2025.00042

    work page doi:10.1109/icdcsw63273.2025.00042 2025

  2. [2]

    The Snowflake Elastic Data Warehouse

    B. Dageville, T. Cruanes, M. Zukowski, V. Antonov, A. Avanes, J. Bock, J. Claybaugh, D. Engovatov, M. Hentschel, J. Huang, A. W. Lee, A. Motivala, A. Q. Munir, S. Pelley, P. Povinec, G. Rahn, S. Triantafyllis, and P. Unterbrunner. “The Snowflake Elastic Data Warehouse.” In Proc. of ACM SIGMOD, 2016

    work page 2016

  3. [3]

    SEE++: Evolving Snowpark Execution Environment for Modern Workloads

    Gaurav Jain et al., “SEE++: Evolving Snowpark Execution Environment for Modern Workloads”. arXiv preprint arXiv:2511.12457

    work page arXiv

  4. [4]

    Drouhard, C

    J. Wang, D. Crawl, S. Purawat, M. Nguyen and I. Altintas, "Big data provenance: Challenges, state of the art and opportunities," 2015 IEEE International Conference on Big Data (Big Data), Santa Clara, CA, USA, 2015, pp. 2509-2516, doi: 10.1109/BigData.2015.7364047

    work page doi:10.1109/bigdata.2015.7364047 2015

  5. [5]

    Online load balancing for MapReduce with skewed data input,

    Y. Le, J. Liu, F. Ergün and D. Wang, "Online load balancing for MapReduce with skewed data input," IEEE INFOCOM 2014 - IEEE Conference on Computer Communications, Toronto, ON, Canada, 2014, pp. 2004-2012, doi: 10.1109/INFOCOM.2014.6848141

    work page doi:10.1109/infocom.2014.6848141 2014

  6. [6]

    Creating Automated Optimizations for Python User -Defined Functions with Snowpark's Parallel Execution - https://www.snowflake.com/en/engineering-blog/snowpark-parallel- python-udf-optimization/

    work page

  7. [7]

    https://www.grpc.io/

    gRPC - A High-Performance, Open-Source Universal RPC Framework. https://www.grpc.io/

    work page

  8. [8]

    The Data Warehouse Toolkit: The Definitive. Guide to Dimensional Modeling. 3rd ed

    R. Kimball, and M. Ross. “The Data Warehouse Toolkit: The Definitive. Guide to Dimensional Modeling. 3rd ed.” Hoboken, NJ: John Wiley & Sons, 2013

    work page 2013

  9. [9]

    https://www.tpc.org/tpcx - bb/default5.asp

    TPCx-BB - A Big Data Benchmark. https://www.tpc.org/tpcx - bb/default5.asp

    work page

  10. [10]

    Devi, D. C. (2016). Load balancing in cloud computing environment using improved weighted round robin algorithm for nonpreemptive dependent tasks. The Scientific World Journal, 2016, 1 –14. https://doi.org/10.1155/2016/3896065

    work page doi:10.1155/2016/3896065 2016

  11. [11]

    Deployment of Query Plans on Multicores

    Giceva, J., et al. (2014). "Deployment of Query Plans on Multicores." Proceedings of the VLDB Endowment (PVLD), 8(3), pp. 233-244

    work page 2014

  12. [12]

    Software Complexity and Software Maintenance Costs

    Sloan, J. J. (1990). "Software Complexity and Software Maintenance Costs." MIT Thesis Archive

    work page 1990

  13. [13]

    Big data platforms: What's next?

    Borkar, V. R., Carey, M. J., & Li, C. (2012). "Big data platforms: What's next?" XRDS: Crossroads, The ACM Magazine for Students

    work page 2012

  14. [14]

    Implementing Fault-Tolerant Services Using the State Machine Approach: A Tutorial

    Schneider, F. B. (1990). "Implementing Fault-Tolerant Services Using the State Machine Approach: A Tutorial." ACM Computing Surveys, 22(4), pp. 299-319

    work page 1990

  15. [16]

    Containerized execution of UDFs: An experimental evaluation

    Saur, K., et al. (2022). "Containerized execution of UDFs: An experimental evaluation." Proceedings of the VLDB Endowment (PVLDB), 15(11)

    work page 2022

  16. [17]

    A Survey of Data Skew Handling in MapReduce

    Li, S., Hu, S., & Li, J. (2015). "A Survey of Data Skew Handling in MapReduce." International Journal of Parallel Programming, 43(3)

    work page 2015