Pith. sign in

REVIEW 4 major objections 5 minor 51 references

AFrame: Extending DataFrames for Large-Scale Modern Data Analysis (Extended Version)

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read AFrame lets Pandas-style dataframes scale on a database backend, translating each operation into a query for AsterixDB.

desk verdict AFrame is a real, useful system paper about giving data scientists a Pandas-like API over AsterixDB, but its headline performance claim is carried by a comparison that charges Spark for file loads while omitting AFrame's own data-ingestion cost. read the letter →

arxiv 1908.06719 v1 pith:A4ZC6VIF submitted 2019-08-19 cs.DB

Phanwadee Sinthong , Michael J. Carey This is my paper · ORCID
classification cs.DB
keywords DataFramesAsterixDBSQL++lazyevaluationdistributedmanagementmicro-benchmarkPandasmachinelearningUDFs
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

AFrame claims that data scientists do not have to abandon the Pandas-style DataFrame workflow when data outgrows a single machine: by implementing that interface on top of a distributed database system, every operation can be translated into a query and executed lazily on managed, indexed data. The paper argues this removes the usual productivity cliff between local exploration and distributed processing, including the step where custom machine learning models are applied to live data. If the claim holds, analysts gain a single-node-like experience on cluster-scale and ever-arriving data, with the database optimizer handling efficiency. The paper backs the claim with a 12-expression micro-benchmark on synthetic data up to 40 GB, reporting that AFrame is competitive with and often faster than Spark DataFrames, especially in total time that includes DataFrame creation.

What carries the argument

The load-bearing mechanism is the translation layer: AFrame keeps a Pandas-like Python API but defers execution, accumulating operations into an internal query representation that is emitted as SQL++ and evaluated by AsterixDB on demand. Lazy evaluation lets a chain of DataFrame steps be optimized as one query, and AsterixDB's open datatypes, secondary indexes, user-defined functions, and live data feeds are exposed through that same interface. The same mechanism lets pre-trained models from tools like Scikit-Learn be registered as UDFs and mapped over columns of distributed data, and lets results be persisted as new datasets without a file round-trip.

What would settle it

Re-run the same 12 expressions on the same hardware but start AFrame from empty AsterixDB datasets and include the time to load the JSON, CSV, or Parquet files and build the indexes; if AFrame's total-time advantage disappears or reverses, the scalability claim would rest on pre-loading rather than on query execution.

Watch

Extended reading notes

Core claim

The central discovery is that a database-backed DataFrame layer can match or beat a compute-engine DataFrame layer on representative analytic operations, despite doing strictly more work by managing, indexing, and persisting data. AFrame achieves this by incrementally building a SQL++ query for every Pandas-like expression and sending it to AsterixDB only when results are needed; because the data already lives in the database, DataFrame creation is just a handle, and operations such as count, filter, range selection, and join can use indexes or the query optimizer. The benchmark's headline finding is that in total elapsed time (creation plus expression), AFrame is fastest on the single node, and in the cluster experiments it remains competitive with Spark's JSON and Parquet DataFrames while beating JSON-based Spark by an order of magnitude on large datasets that force disk spilling.

Load-bearing premise

The headline comparisons assume that the 12-expression benchmark is representative of real DataFrame work and that crediting AFrame for already-loaded, indexed data while Spark pays for reading files does not unfairly favor AFrame.

Editorial extensions

If this is right

  • If a database backend can absorb DataFrame operations, moving from a laptop to a cluster no longer requires rewriting analysis code or learning a new engine's API.
  • Because the data is managed and indexed, repeated exploratory operations avoid repeated full file scans, so interactive question-answer cycles stay fast as datasets grow.
  • Custom machine learning models trained locally can be applied to distributed and live data through a simple map call, without a separate distributed machine-learning framework.
  • The benchmark's separation of DataFrame-creation time from expression time gives data scientists a way to see whether their bottleneck is loading or computing, and the workload can be extended with more expressions and frameworks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the paper leaves implicit is that the same translation-layer design could be ported to other SQL or query engines, making DataFrame code a portable abstraction over whatever database sits underneath.
  • The total-time advantage depends on data being resident and indexed in the database; a fairer long-run comparison would also measure the cost of getting data into AsterixDB in the first place, including feed ingestion and index maintenance.
  • The benchmark's synthetic data stresses uniform predicates and simple schemas; real workloads with nested JSON, skewed distributions, or window functions may shift the ranking, so AFrame's advantage is demonstrated for this workload class rather than proven for all analytics.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents AFrame, a Pandas-like DataFrame library that executes operations on top of Apache AsterixDB by incrementally translating DataFrame expressions into SQL++ queries. The system is designed to provide a scale-independent user experience for data scientists: lazy evaluation, database-side query optimization, indexing, support for user-defined machine learning models via AsterixDB UDFs, and operation on live ingested data. The paper also proposes a Wisconsin-inspired micro-benchmark of twelve DataFrame operations and reports single-node and multi-node experiments comparing Pandas, Pandas on Ray (Modin), three Spark DataFrame variants (JSON, JSON with schema, Parquet), and three AFrame variants (open datatype, schema, indexed). The central claim is that AFrame transparently scales DataFrame operations and machine learning model invocation, and that its initial performance is competitive with, and often faster than, Spark DataFrames on the tested workloads.

Significance. If the performance claims were established, the paper would make a useful contribution: it demonstrates a concrete architecture for integrating DataFrame APIs with a full big-data management system, and the micro-benchmark separates DataFrame creation time from expression execution time, which is a good practice for understanding the components of DataFrame workload costs. The use of AsterixDB's indexing, query optimization, and UDF support is a substantive differentiator from file-based DataFrame engines, and the paper's focus on managed, live, and indexed data is a legitimate design point. However, the headline performance comparison is currently not established because the total-time metric measures the two systems under different workload assumptions, and the experimental appendix does not provide the reproducibility details (versions, error bars, artifact) that would let a reader assess the reported differences. The qualitative architectural claims are sound and the raw numbers in the appendix are internally consistent, but the quantitative 'often faster' conclusion needs reframing and additional evidence.

major comments (4)
  1. [Section IV-C, Appendix Section D, Tables V-VI] The total-time comparison is confounded by asymmetric data preparation: AFrame's data is already resident and indexed in AsterixDB, while Spark's total time includes reading files and schema inference; the paper never reports AsterixDB bulk-load and index-build costs, so the claim that AFrame is 'often faster' in total time is not established for a one-shot workload starting from raw files.
  2. [Section IV-D, Tables V-VIII] The experimental reproducibility details are insufficient: no software versions, no artifact link, and no variance information are reported for the averages in Tables V-VIII, so the reader cannot assess whether the close performance differences are meaningful.
  3. [Section IV-C, Section V-A, Table VI] The AFrame Index variant is compared against Spark without any equivalent index or preprocessed structure, yet the paper uses these comparisons to support general 'AFrame was fastest' statements; the indexed and unindexed cases should be reported and interpreted separately.
  4. [Section V-B, Tables III-IV] The multi-node speedup experiment is confounded by the single-node Spark JSON disk-spill effect, so the reported 'speedup' from 1 to 2 nodes does not cleanly measure parallel scaling; the paper should separate the spill-elimination effect from true speedup and report variance.
minor comments (5)
  1. [Appendix Tables V-VIII] The raw-data tables are difficult to use because the column headers do not identify which system each column belongs to; please reformat with explicit multi-level headers.
  2. [Figures 8-15] Several figure panels repeat the same legend and the asterisk notation appears only in some captions; please use shared legends and define all annotations consistently.
  3. [Section IV-B, Section IV-C] Pandas on Ray is evaluated with CSV input while all other systems use JSON or Parquet; this format inconsistency should be highlighted in the benchmark definition as well as in the setup narrative.
  4. [Section IV-D] The warm-up exclusion rationale mentions JVM overhead but is applied to all systems including Pandas; please clarify the procedure for non-JVM systems.
  5. [General] Please provide a URL or repository identifier for AFrame and the benchmark code, since the paper frames both as reusable contributions.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark claims rest on direct external comparisons and measured execution times, not on fitted inputs or self-referential definitions.

full rationale

The paper's central claims are about the design and measured performance of AFrame, not about a derived quantity that reduces to its inputs. AFrame's run times are obtained by executing the twelve benchmark expressions against real systems (Spark, Pandas, Pandas on Ray, and AFrame), and the benchmark queries and datasets are adapted from the external Wisconsin Benchmark tradition. No parameter is fitted to a subset of the data and then reported as a prediction, and no claimed result is true by construction of the benchmark definitions. Citations to AsterixDB and related AsterixDB work describe the underlying platform capabilities; they provide context for AFrame's implementation but are not the evidence that AFrame is competitive, which comes from the reported experiments. The most serious concern is that AFrame's total-time figures assume data already resident and indexed in AsterixDB while Spark's total time includes reading files; this is a benchmark fairness and correctness issue, not circularity, because the measured AFrame times are still independent outputs of actual query execution rather than consequences of the benchmark's assumptions.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters (fitted or hand-tuned) are needed because the paper is an empirical systems study. The listed axioms are the key background assumptions on which the performance and functionality claims rest. No new physical or conceptual entities are introduced; AFrame is a software system, not a postulated theoretical entity.

assumptions (3)
  • domain assumption AsterixDB's SQL++ runtime and optimizer correctly and efficiently execute the queries generated by AFrame.
    The whole claim depends on the correctness and efficiency of the backend; relied on in Section III-B (Operating on Data).
  • domain assumption The 12 benchmark expressions are representative of common exploratory DataFrame operations.
    Used to generalize from the benchmark to real workloads; introduced in Section IV-B.
  • domain assumption The hardware environment (EC2 m4.large, 2 cores, 8GB memory) and the Wisconsin-style synthetic data are sufficient to draw conclusions about performance on modern data.
    All experiments use this environment; Section IV-D.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AFrame: Extending DataFrames for Large-Scale Modern Data Analysis (Extended Version)." pith.science (2026). https://pith.science/paper/A4ZC6VIF

@misc{pith2026190806719,
  author       = {Pith},
  title        = {Pith review of: AFrame: Extending DataFrames for Large-Scale Modern Data Analysis (Extended Version)},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/A4ZC6VIF}},
  note         = {Machine review of arXiv:1908.06719}
}
read the original abstract

Analyzing the increasingly large volumes of data that are available today, possibly including the application of custom machine learning models, requires the utilization of distributed frameworks. This can result in serious productivity issues for "normal" data scientists. This paper introduces AFrame, a new scalable data analysis package powered by a Big Data management system that extends the data scientists' familiar DataFrame operations to efficiently operate on managed data at scale. AFrame is implemented as a layer on top of Apache AsterixDB, transparently scaling out the execution of DataFrame operations and machine learning model invocation through a parallel, shared-nothing big data management system. AFrame incrementally constructs SQL++ queries and leverages AsterixDB's semistructured data management facilities, user-defined function support, and live data ingestion support. In order to evaluate the proposed approach, this paper also introduces an extensible micro-benchmark for use in evaluating DataFrame performance in both single-node and distributed settings via a collection of representative analytic operations. This paper presents the architecture of AFrame, describes the underlying capabilities of AsterixDB that efficiently support modern data analytic operations, and utilizes the proposed benchmark to evaluate and compare the performance and support for large-scale data analyses provided by alternative DataFrame libraries.

Figures

Figures reproduced from arXiv: 1908.06719 by the authors.

Figure 1
Figure 1. shows an example of creating an open datatype ‘Tweet’ with only the field ‘id’ being pre-defined and two datasets called ‘TrainingData’ and ‘LiveTweets’ which store records of this Tweet datatype. The TrainingData dataset is populated by reading data from a local file system. In this example, it is being populated using a labeled airline sentiment dataset. AsterixDB also provides support for user-defined functions (… view at source ↗
Figure 2
Figure 2. Initializing AFrame Objects B. Operating on Data As most EDA tools are designed to work with in-memory data, the eager evaluation strategy can suffice even when a session involves multiple scans over the entire dataset. How￾ever, multiple scans over a large distributed dataset would be very costly and have a negative effect on system performance. AFrame leverages lazy evaluation. AFrame operations are incrementally … view at source ↗
Figure 4
Figure 4. Training a Scikit-Learn Pipeline is a dataset, publicly available on Kaggle [10], containing Twitter posts related to users’ experiences with U.S. airlines released by CrowdFlower [7]. The dataset contains labeled tweet sentiments which are positive, negative, and neutral. The first step in [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figures from the paper (13 more)
Figure 2
Figure 2. Figure 2: Input 11 applies the pre-trained Stanford CoreNLP [PITH_FULL_IMAGE:figures/full_fig_p005_2.png]
Figure 5
Figure 5. Figure 5: Applying CoreNLP and Scikit-Learn models [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 7
Figure 7. Figure 7: Scalable Wisconsin benchmark: attributes [23] [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: XS Results of Single Node Evaluation evaluation metrics. Spark produces different runtime plans for the JSON-based DataFrame and the Parquet-based DataFrame, resulting in the difference in their task execution times even after the schema inferencing step. AFrame was th…
Figure 9
Figure 9. Figure 9: Single Node Evaluation: Expression 1-5 Results (* = value where the bar ends) [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 10
Figure 10. Figure 10: Single Node Evaluation: Expression 6-10 Results (* = value where the bar ends) [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]
Figure 11
Figure 11. Figure 11: Single Node Evaluation: Expression 11-12 Results (* = value where the bar ends) [PITH_FULL_IMAGE:figures/full_fig_p012_11.png]
Figure 12
Figure 12. Figure 12: Multi-Node Speedup Evaluation Results [PITH_FULL_IMAGE:figures/full_fig_p013_12.png]
Figure 13
Figure 13. Figure 13: Multi-Node Speedup Evaluation Results (continued) [PITH_FULL_IMAGE:figures/full_fig_p014_13.png]
Figure 14
Figure 14. Figure 14: Multi-Node Scaleup Evaluation Results [PITH_FULL_IMAGE:figures/full_fig_p015_14.png]
Figure 15
Figure 15. Figure 15: Multi-Node Scaleup Evaluation Results (continued) [PITH_FULL_IMAGE:figures/full_fig_p016_15.png]
Figure 16
Figure 16. Figure 16: Create a Twitter feed to collect tweets. [PITH_FULL_IMAGE:figures/full_fig_p018_16.png]
Figure 17
Figure 17. Figure 17: Create datatypes and datasets to use for benchmarking [PITH_FULL_IMAGE:figures/full_fig_p018_17.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 51 canonical work pages

  1. [1]

    https://wesmckinney.com/blog/apache-arrow-pandas-internals/

    Apache Arrow and the 10 Things I Hate About pandas. https://wesmckinney.com/blog/apache-arrow-pandas-internals/

    work page

  2. [2]

    https://asterixdb.apache.org/

    Apache asterixdb. https://asterixdb.apache.org/

    work page

  3. [3]

    https://hive.apache.org/

    Apache Hive. https://hive.apache.org/

    work page

  4. [4]

    https://parquet.apache.org/

    Apache Parquet. https://parquet.apache.org/

    work page

  5. [5]

    http://spark.apache.org/

    Apache Spark. http://spark.apache.org/

    work page

  6. [6]

    http://tez.apache.org/

    Apache Tez. http://tez.apache.org/

    work page

  7. [7]

    http://www.crowdflower.com/

    CrowdFlower. http://www.crowdflower.com/

    work page

  8. [8]

    http://dask.org/

    Dask. http://dask.org/

    work page

Show all 51 references
  1. [9]

    http://sortbenchmark.org/

    GraySort benchmark. http://sortbenchmark.org/

  2. [10]

    http://www.kaggle.com/crowdflower/twitter-airline-sentiment/

    Kaggle. http://www.kaggle.com/crowdflower/twitter-airline-sentiment/

  3. [11]

    https://modin.readthedocs.io/en/latest/

    Modin. https://modin.readthedocs.io/en/latest/

  4. [12]

    http://pandas.pydata.org/

    Pandas. http://pandas.pydata.org/

  5. [13]

    https://rise.cs.berkeley.edu/blog/pandas-on-ray-early- lessons/

    Pandas on Ray. https://rise.cs.berkeley.edu/blog/pandas-on-ray-early- lessons/

  6. [14]

    http://www.r-project.org/

    R. http://www.r-project.org/

  7. [15]

    http://developer.twitter.com/

    Twitter API. http://developer.twitter.com/

  8. [16]

    Abadi et al

    M. Abadi et al. Tensorflow: A system for large-scale machine learn- ing. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI) , pages 265–283, 2016

    2016

  9. [17]

    Alkowaileet et al

    W. Alkowaileet et al. End-to-end machine learning with Apache Aster- ixDB. In Proceedings of the Second Workshop on Data Management for End-To-End Machine Learning , page 6. ACM, 2018

    2018

  10. [18]

    Alsubaiee et al

    S. Alsubaiee et al. AsterixDB: a scalable, open source BDMS. PVLDB, 7(14):1905–1916, 2014

    1905

  11. [19]

    Armbrust et al

    M. Armbrust et al. Scaling Spark in the real world: performance and usability. PVLDB, 8(12):1840–1843, 2015

    2015

  12. [20]

    Armbrust et al

    M. Armbrust et al. Spark SQL: Relational data processing in Spark. In SIGMOD, pages 1383–1394, 2015

    2015

  13. [21]

    Chamberlin

    D. Chamberlin. SQL++ for SQL Users: A Tutorial. September 2018. Available via Amazon.com

    2018

  14. [22]

    B. F. Cooper et al. Benchmarking cloud serving systems with YCSB. In ACM SoCC, pages 143–154, 2010

    2010

  15. [23]

    D. J. DeWitt. The Wisconsin benchmark: Past, present, and future. In J. Gray, editor, The Benchmark Handbook . Morgan Kaufmann, 1993

    1993

  16. [24]

    Ghazal et al

    A. Ghazal et al. BigBench: towards an industry standard benchmark for big data analytics. In SIGMOD, pages 1197–1208, 2013

    2013

  17. [25]

    Kluyver et al

    T. Kluyver et al. Jupyter Notebooks-a publishing format for reproducible computational workflows. In Proceedings of the 20th International Conference on Electronic Publishing (ELPUB) , pages 87–90, 2016

    2016

  18. [26]

    Liu et al

    X. Liu et al. Smart meter data analytics: systems, algorithms, and bench- marking. ACM Transactions on Database Systems (TODS) , 42(1):2, 2017

    2017

  19. [27]

    McKinney et al

    W. McKinney et al. Data structures for statistical computing in Python. In Proceedings of the 9th Python in Science Conference , volume 445, pages 51–56. Austin, TX, 2010

    2010

  20. [28]

    Meijer, B

    E. Meijer, B. Beckman, and G. Bierman. LINQ: Reconciling objects, relations and XML in the .NET framework. In SIGMOD, pages 706– 706, 2006

    2006

  21. [29]

    Meng et al

    X. Meng et al. MLlib: machine learning in Apache Spark. Journal of Machine Learning Research , 17(1):1235–1241, 2016

    2016

  22. [30]

    TDSP: Team Data Science Process

    Microsoft. TDSP: Team Data Science Process. 2017

    2017

  23. [31]

    Moritz et al

    P. Moritz et al. Ray: A distributed framework for emerging AI applications. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI) , pages 561–577, 2018

    2018

  24. [32]

    R. O. Nambiar and M. Poess. The making of TPC-DS. In PVLDB, pages 1049–1058, 2006

    2006

  25. [33]

    Pedregosa et al

    F. Pedregosa et al. Scikit-learn: machine learning in Python. Journal of Machine Learning Research , 12(10):2825–2830, 2011

    2011

  26. [34]

    D. A. Schmidt. The structure of typed programming languages . MIT press, 1994

    1994

  27. [35]

    Shvachko et al

    K. Shvachko et al. The Hadoop distributed file system. In IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST) , pages 1–10, 2010

    2010

  28. [36]

    Socher et al

    R. Socher et al. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 1631–1642, 2013

    2013

  29. [37]

    J. W. Tukey. Exploratory Data Analysis: Limited Preliminary Ed . Addison-Wesley Publishing Company, 1970

    1970

  30. [38]

    Venkataraman et al

    S. Venkataraman et al. SparkR: scaling R programs with spark. In SIGMOD, pages 1099–1104, 2016

    2016

  31. [39]

    adapter-name

    M. Zaharia et al. Apache Spark: a unified engine for big data processing. Communications of the ACM , 59(11):56–65, 2016. APPENDIX A. AsterixDB Twitter Feed CREATE FEED TwitterFeed with { "adapter-name" : "push_twitter", "type-name" : "Tweet", "format" : "twitter-status", "cons...

    2016

  32. [40]

    SELECT VALUE COUNT( *) FROM Data

  33. [41]

    SELECT t.two, t.four FROM Data t LIMIT 5

  34. [42]

    SELECT VALUE COUNT( *) FROM Data t WHERE t.ten = x AND t.twentyPercent = y AND t.two = z

  35. [43]

    SELECT grp_id, COUNT( *) AS cnt FROM Data t GROUP BY t.oddOnePercent AS grp_id

  36. [44]

    SELECT VALUE UPPER(t.stringu1) FROM Data t LIMIT 5

  37. [45]

    SELECT MAX(t.unique1) FROM Data t

  38. [46]

    SELECT MIN(t.unique1) FROM Data t

  39. [47]

    SELECT grp_id, MAX(t.four) AS max FROM Data tGROUP BY t.twenty AS grp_id

  40. [48]

    SELECT VALUE t FROM Data t ORDER BY t.unique1 DESC LIMIT 5

  41. [49]

    SELECT VALUE t FROM Data t WHERE t.ten = x LIMIT 5

  42. [50]

    SELECT VALUE t FROM Data t WHERE t.onePercent >= x AND t.onePercent <= y

  43. [51]

    SELECT VALUE COUNT( *) FROM (SELECT l,r FROM leftData l JOIN rightData r ON l.unique1 = r.unique1) t; D. Benchmark Timing Points • Pandas & Pandas on Ray Timing # DataFrame creation time df = pd.read_json(file_path) # Expression-only time df.head() • Spark Timing # DataFrame c...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.