REVIEW 4 cited by
Towards Scalable Dataframe Systems
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Dataframes are a popular abstraction to represent, prepare, and analyze data. Despite the remarkable success of dataframe libraries in Rand Python, dataframes face performance issues even on moderately large datasets. Moreover, there is significant ambiguity regarding dataframe semantics. In this paper we lay out a vision and roadmap for scalable dataframe systems. To demonstrate the potential in this area, we report on our experience building MODIN, a scaled-up implementation of the most widely-used and complex dataframe API today, Python's pandas. With pandas as a reference, we propose a simple data model and algebra for dataframes to ground discussion in the field. Given this foundation, we lay out an agenda of open research opportunities where the distinct features of dataframes will require extending the state of the art in many dimensions of data management. We discuss the implications of signature data-frame features including flexible schemas, ordering, row/column equivalence, and data/metadata fluidity, as well as the piecemeal, trial-and-error-based approach to interacting with dataframes.
Forward citations
Cited by 4 Pith papers
-
Learning Lineage Constraints for Data Science Operations
XProv is a proposed architecture that uses lineage-constraint tags and small-sample learning to capture cross-library data lineage with uncertainty.
-
Metadata Management for AI-Augmented Data Workflows
TableVault is a metadata governance framework that records ingestion events, operation status, execution parameters, and lineage for human-AI data workflows, demonstrated on a document classification case study.
-
TableVault: Managing Dynamic Data Collections for LLM-Augmented Workflows
TableVault describes a system design for managing versioned, reproducible dataframe collections in LLM-augmented workflows, but it ships no implementation or evaluation.
-
Fast Capture of Cell-Level Provenance in Numpy
A C-backed numpy data type with preallocated provenance buffers captures cell-level lineage up to 275x faster than a naive Python implementation on element-wise operations and scales to 100 million cells.
Discussion (0). Sign in to comment.