{"id":"cc7543a9-0aea-4bba-8432-51477cbfb165","arxiv_id":"2506.02345","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"PandasBench, a 102-notebook executable Pandas API benchmark, shows that Modin, Dask, and Koalas fail on most real-world notebooks and that Dias, while faster, sometimes rewrites code incorrectly.","lead":"PandasBench is a new executable benchmark built from 102 real Kaggle notebooks and 3,721 Pandas-API cells, designed to give dataframe systems a realistic performance and coverage yardstick. When applied to Modin, Dask, Koalas, and Dias, it shows most popular Pandas replacement engines fail on most real-world notebooks, and that earlier speedup claims came from easier, synthetic workloads.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Manual preparation is never validated for semantic or performance equivalence, so the central 'real-world notebook' claim rests on an unaudited transformation.","rationale":"The reader's weakest-assumption analysis correctly identifies that the manual preparation pipeline is the load-bearing link between 'real-world notebooks' and 'PandasBench'. My stress-test adds two concrete sharpenings. First, the cleaning rules in Appendix B are not merely cosmetic: they explicitly delete Pandas code that depends on non-Pandas libraries and remove downstream uses of deleted dataframe-column assignments. This can change which Pandas operations are benchmarked, not just how long they take. Second, the paper gives no quantitative evidence about the size or effect of the edits: no counts of modified/deleted cells, no before/after API trace comparison, no attempt to show that the cleaned notebook computes the same pandas-relevant results as the original. The reported discrepancy between 102 downloaded and 114 fixed/cleaned, with 12 discarded for undocumented reasons, reinforces that the construction path has not been audited. The concern is empirical and testable: the artifact is public, so an independent re-preparation and differential test would settle whether the edits changed the workload. If the differential test shows the pipeline is faithful, the central claim stands. If it shows divergence, the benchmark still has value as a curated resource, but the 'real-world coverage' claim would need to be weakened. The reader's CONDITIONAL verdict already reflects this uncertainty, so I do not recommend changing the verdict. I also note that the paper has independent support in the form of a released artifact and a detailed written preparation process, which makes the proposed audit feasible; my concern is about evidence, not intent.","tokens_in":21769,"tokens_out":5385,"duration_ms":61017,"concrete_test":"Independently re-run the preparation process, blindly, on a random sample of roughly 20 notebooks from the original Kaggle pool using only the §4.5/Appendix B instructions and the released artifact, then differential-test each independently prepared notebook against the released PandasBench version: compare cell-level Pandas API call traces and final outputs. If the independent preparation differs from the released benchmark on more than a small threshold (e.g., 10% of cells in any sampled notebook), the manual pipeline is not reproducible and the real-world claim is not supported. Record the reasons for the 12 discarded notebooks; if any reason correlates with a specific technique's failures, recompute the denominators.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that PandasBench measures real-world Pandas code and that prior techniques fail or slow down on that code. The load-bearing premise is that the manual preparation pipeline in §4.5 and Appendix B preserves the semantics and performance-relevant behavior of the original Kaggle notebooks. That premise is asserted but never checked. The pipeline deletes code that depends on non-Pandas libraries 'even if it includes Pandas code', removes following uses of columns whose assignments were deleted, replaces hardcoded values with runtime values, inserts evaluator calls for lazy evaluation, and scales inputs. Each of these steps can change which Pandas calls execute, how they execute, or what data they see. The paper reports no differential testing between original and prepared notebooks, no edit-size statistics, and no independent audit of the manual fixes. The count inconsistency between §3 (102 downloaded) and §4 (114 fixed/cleaned, 12 discarded), plus the undocumented reasons for the 12 discards, makes the retention path opaque. If edits removed or altered even a modest fraction of Pandas computations, the 3,721-cell benchmark would not represent the original real-world workload, and both the coverage counts and speedup results could shift. This is not an accusation that the edits are wrong; it is a missing validation step for a benchmark whose entire purpose is representativeness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"PandasBench is presented as a benchmark suite of 102 real-world Kaggle notebooks (3,721 cells) for the Pandas API. The paper proposes four requirements for such a benchmark—real-world coverage, relevance of code, benchmark appropriateness, and data scaling—and argues that no existing collection satisfies them. The authors describe a manual pipeline that fixes broken code, cleans out non-Pandas code, adapts notebooks for repeated execution, and scales input data non-uniformly. They use the benchmark to evaluate Modin, Dask, Koalas, and Dias, reporting low real-world coverage for the alternatives (Modin 72/102, Dask 3–4, Koalas 10) and mostly slowdowns relative to pandas, with Dias the only technique achieving speedups on a substantial subset (up to 55 notebooks). The headline conclusion is that prior constrained benchmarks overstate the speedups of Pandas alternatives.","tokens_in":21967,"tokens_out":5586,"duration_ms":50216,"significance":"If correct, PandasBench would be the first large, executable, real-world benchmark for the Pandas API, and the negative results for Modin, Dask, and Koalas would be an important correction to prior evaluations based on small or synthetic workloads. The paper's strengths include a publicly available artifact, a detailed description of the preparation pipeline, dynamic API coverage analysis, a non-uniform scaling infrastructure, and the largest head-to-head evaluation of these techniques to date. These are substantial empirical contributions. The main limitation is that the manual preparation pipeline's fidelity to the original notebooks is asserted but not validated; this currently weakens the benchmark's representativeness claim and, in turn, the strength of the negative results.","major_comments":[{"comment":"The sampling and retention narrative is internally inconsistent. §3 states that 102 notebooks were downloaded randomly, while §4 states that 'we fixed and cleaned 114 notebooks and discarded 12 notebooks after applying the process in §4.5.' These numbers do not reconcile: if 102 is the final count, the pre-discard count would have to be 114, meaning 12 additional notebooks are unaccounted for in §3; alternatively, if 102 is the download count, fixing and cleaning 114 is impossible. The criteria for discarding the 12 notebooks are also never specified. Since the paper's central claim is a random sample of real-world notebooks, the reader must be able to trace the exact path from initial download to final benchmark; without this, the randomness and representativeness claims are unverifiable.","section":"§3 and §4"},{"comment":"The manual preparation pipeline is the load-bearing premise for the claim that PandasBench evaluates real-world Pandas code, but the paper provides no validation that the edited notebooks preserve the semantics and performance-relevant behavior of the originals. The pipeline explicitly deletes code that depends on non-Pandas libraries 'even if it includes Pandas code', removes following uses of columns whose assignments were deleted, replaces hardcoded values with runtime values, inserts evaluate_eager calls for lazy evaluation, and scales inputs. Each of these steps can change which Pandas operations execute, how they execute, or what data they observe. The paper reports no differential testing between original and prepared notebooks, no edit-size statistics (e.g., the fraction of cells or API calls removed or altered per notebook), and no independent audit of the manual fixes. I am not accusing the edits of being incorrect; rather, this is a missing validation step for a benchmark whose entire purpose is representativeness. The authors should provide evidence such as before/after API-call coverage comparisons on a sample, a per-notebook log of edits, or an external audit of the preparation process.","section":"§4.5 and Appendix B"},{"comment":"The abstract reports that Dask ran 4 notebooks (~4%), while §6.2 states that Dask 'could run only 3 (~3%)' and Figure 3a shows 3. One of these is incorrect. This inconsistency affects one of the paper's headline claims—that the most notebooks that got a speedup were 8/102 for Modin and 0 for both Koalas and Dask—because the Dask coverage count is part of that comparative statement. Please correct the discrepancy and audit all other reported counts (including cell-level coverage figures) for consistency.","section":"Abstract and §6.2"},{"comment":"The contribution bullet describes the benchmark as composed of 'real-world, randomly picked notebooks.' This overstates the sampling procedure: §3 applies explicit filters—at least 20% of static calls from the Pandas API, at least 10 cells, and exclusion of all 20 Dias notebooks—and the paper later discards 12 more notebooks during preparation. The sample is therefore a filtered random sample of Kaggle notebooks satisfying stated criteria, not a random sample of Kaggle notebooks generally. Without this qualification, the diversity claim (RWC3) is difficult to evaluate, and the reader may mistakenly conclude the benchmark represents all Kaggle notebooks. The wording should be changed to describe the actual selection procedure.","section":"§1 contribution bullet and §3"}],"minor_comments":[{"comment":"The abstract contains a typo: 'Pandas API enchmark' should be 'Pandas API benchmark.'","section":"Abstract"},{"comment":"The final paragraph states 'This process was done automatically although in principle it could be automated.' This is self-contradictory; presumably the intended wording is that the process was done manually although in principle it could be automated. Please fix.","section":"§6.1.5"},{"comment":"The formula for the geometric mean memory+disk ratio is typeset incorrectly: the expression 'GeoMean {...}' with the fraction 'I_T,R / I_pandas,R' is garbled and lacks proper subscripts and delimiters. It should be written as the geometric mean over notebooks in N_{T,R} of I_{T,R}(i) / I_{pandas,R}(i). Please correct the equation.","section":"§6.3.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript would benefit from an external audit of the preparation pipeline, given that the authors are also the creators of Dias, the technique that performs best in the evaluation. The artifact is publicly available, so such an audit is feasible and would substantially strengthen confidence in the results. I would also encourage the editor to consider requiring a clearer disclosure of this relationship in the experimental section, even though the exclusion of Dias's own notebooks is a reasonable step toward reducing bias."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version. PandasBench fills a real gap: it is the first executable, large, real-world Pandas API benchmark, and the evaluation of Modin, Dask, Koalas, and Dias is the largest and most realistic to date. The headline findings—Modin runs 72/102 notebooks, Dask 4/102, Koalas 10/102, Dias 97/102, and most notebooks slow down—are important and, I think, directionally correct. But the benchmark's central claim to represent real-world Pandas code rests on a manual preparation pipeline that is never validated for semantic or performance equivalence, and there is a notebook-count discrepancy that needs explanation.\n\nWhat the paper does well: it identifies the requirements (real-world coverage, relevant code, benchmark appropriateness, data scaling) and makes a persuasive case that prior collections fail them. The non-uniform scaling to target runtimes is a genuine methodological addition. The authors exclude Dias's own notebooks to reduce bias, and they report cell-level and notebook-level coverage plus memory and disk usage. The artifact is available. The dynamic coverage measurement is careful.\n\nSoft spots, in proportion. The main one is the preparation process in Section 4.5 and Appendix B. The process deletes pandas code that depends on non-pandas libraries, removes following uses of columns whose assignments were deleted, replaces hardcoded values with runtime values, inserts evaluator calls for lazy evaluation, and scales inputs. Each step can change which pandas calls execute, how they execute, or what data they see. The paper reports no differential testing between original and prepared notebooks, no edit-size statistics, and no independent audit. I don't doubt the edits were made in good faith, but for a benchmark whose purpose is representativeness, this is an assertion, not a demonstrated property. The count inconsistency (102 downloaded in Section 3 vs. 114 fixed and cleaned in Section 4, with 12 discarded) makes the retention path opaque, and the reasons for the 12 discards are undocumented. Performance numbers are reported without repeated runs or error bars; the scale-factor convergence sentence has a likely typo ('automatically although in principle it could be automated'). These are fixable, but they matter.\n\nIs the central claim still standing? Largely yes. Even if the edits removed some pandas computations, the coverage gap and the lack of speedups on whole notebooks are so large that the qualitative conclusions are unlikely to flip. But the exact numbers, and the 'real-world coverage' claim, need an audit to be fully credible.\n\nWho this is for: anyone evaluating dataframe systems, and anyone using Modin, Dask, Koalas, or Dias in production. Worth a serious referee. I'd recommend sending to peer review, with a request for an audit of the preparation process, documentation of discards, and a fix of the count.","headline":"PandasBench is the first credible large real-world Pandas benchmark and its coverage results are important, but the unvalidated manual cleaning pipeline and a notebook-count inconsistency undermine the representativeness claim until audited.","tokens_in":22575,"tokens_out":3461,"would_cite":true,"duration_ms":31327,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces PandasBench, the first Pandas API benchmark built from 102 real Kaggle notebooks, and shows that on it most Pandas optimization techniques slow code down rather than speed it up.","keywords":["Pandas API","benchmark","real-world coverage","notebook workloads","data scaling","Modin","Dask","Koalas"],"falsifier":"Take the released notebooks with their original Kaggle inputs, run them as downloaded, and compare the set of Pandas API calls, column types, and row counts against the cleaned versions; any material difference in what executes would show PandasBench is not measuring the original real-world code.","tokens_in":21511,"feed_emoji":"🐼","tokens_out":11584,"duration_ms":91698,"temperature":0.7,"pith_summary":"This paper argues that no usable benchmark exists for the Pandas API, and that existing benchmarks from SQL or array computing cannot be borrowed. To fix that, it introduces PandasBench: 102 randomly selected, executable Kaggle notebooks (3,721 cells) that were manually fixed, cleaned of non-Pandas code, adapted for repeated runs, and given per-notebook non-uniform input scaling. The paper uses PandasBench to evaluate four Pandas optimization techniques. It reports that Modin speeds up only 8 of 102 notebooks (~8%), Dask and Koalas speed up none, and Dias speeds up about half while also rewriting some code incorrectly. If these results hold, the speedups reported in earlier constrained evaluations do not transfer to whole real-world Pandas workloads.","feed_headline":"Real notebooks: Modin helps just 8%, Dask and Koalas 0%","feed_subtitle":"A 102-notebook Kaggle benchmark shows prior speedup claims came from small synthetic tests.","key_machinery":"The load-bearing object is the benchmark construction pipeline: random download of 102 Kaggle notebooks with at least 10 cells and at least 20% Pandas API calls; a manual stage that fixes library-version and input-code mismatches, cleans out machine-learning and plotting code while preserving dependent Pandas code, adapts notebooks for repeated execution and eager evaluation of lazy results; and a scaling stage that duplicates or removes rows per notebook to hit a target runtime instead of applying one uniform factor to all data. The paper's definition of real-world coverage (RWC1-RWC3) and the four requirements RWC, REL, APPR, and DSC are the criteria that distinguish PandasBench from prior collections.","core_discovery":"The paper's central claim is that real-world coverage — code that is real, executable, large, and diverse — is the necessary standard for any Pandas API benchmark, and that PandasBench is the first collection satisfying all four requirements it defines: real-world coverage, relevant code only, benchmark-appropriate execution, and effective data scaling. On this benchmark, the paper finds that popular Pandas alternatives largely fail or slow down: Modin runs 72/102 notebooks (~70%) and speeds up 8/102 (~8%) at best; Dask runs 4 notebooks and speeds up none; Koalas runs 10 and speeds up none; Dias runs 97 but produces incorrect rewrites in cases prior work missed. The paper also finds that scaling matters: some notebooks fail only after upscaling (Koalas out-of-memory, Modin internal errors), and one Dask notebook succeeds only after downscaling because a problematic float is cut away. This is presented as evidence that prior speedups came from constrained, often synthetic or tiny test sets.","pith_inferences":["Beyond the paper: because the benchmark scales each notebook to a target runtime, the published speedups may understate how these techniques behave on the original, often much smaller Kaggle inputs, where startup overheads make up a larger share of runtime.","Beyond the paper: the failure taxonomy (missing method, missing argument, unsupported use case, invalid output, bad type inference, scaling error) could be turned into a compatibility regression suite for Pandas API implementations.","Beyond the paper: the finding that about 10% of input files stress type inference, plus a notebook producing a transposed frame with millions of columns, suggests targeted microbenchmarks for inference engines and transpose-heavy workloads would complement PandasBench.","Beyond the paper: a direct test of reusability would be whether two independent teams following the same manual preparation rules converge on similar notebooks and close to the same coverage counts."],"forward_implications":["Prior favorable evaluations of Modin, Dask, and Koalas on small or synthetic Pandas code do not predict whole-notebook behavior: on PandasBench, 92% or more of real notebooks are not sped up.","A Pandas alternative cannot act as a drop-in replacement if it fails to run 30% (Modin) to 96% (Dask) of real notebooks.","Benchmark scaling is not neutral: coverage and relative performance shift with input size, so a useful Pandas benchmark must support per-notebook, non-uniform scaling and report results across target runtimes.","Techniques that rewrite Pandas code (Dias) need correctness checks on real workloads, since PandasBench found incorrect rewrites that earlier, smaller evaluations did not expose.","PandasBench provides a reusable 102-notebook suite with data and harness so future Pandas API techniques can report coverage, speedup, and memory/disk usage on real-world code."],"supporting_citations":[{"why":"Supplies the prior 20-notebook Dias evaluation and the observation that Modin slows down real-world snippets, which PandasBench extends.","marker":"[4]"},{"why":"Modin's original evaluation used four synthetic single-operation benchmarks, the constrained setting whose speedups PandasBench contrasts.","marker":"[28]"},{"why":"Dask is one of the three Pandas alternatives evaluated; Appendix A uses Dask's TPC-H behavior to argue relational benchmarks do not transfer.","marker":"[21]"},{"why":"Koalas is one of the three Pandas alternatives evaluated, with failures attributed to missing arguments and JVM memory limits.","marker":"[25]"},{"why":"Identifies dataframe-specific challenges (schema inference, unbounded column cardinality, transposition) used in Section 5.2 to show PandasBench reaches dataframe-unique territory.","marker":"[27]"},{"why":"The TPC benchmark family is the main candidate for repurposing, and the paper argues none of its variants fit the Pandas API.","marker":"[42]"},{"why":"The TPC-H specification is cited for uniform scaling and the old-sales refresh function, the contrasts to PandasBench's non-uniform scaling and data cleaning.","marker":"[41]"},{"why":"The million-notebook analysis motivates Pandas's importance by finding that 42.3% of notebooks import it.","marker":"[34]"},{"why":"Kaggle is the source of the real-world notebooks and provides the original data needed for executability.","marker":"[17]"},{"why":"Pandera is the closest known way to generate valid input data but requires schemas, supporting the choice to collect notebooks with their original data.","marker":"[3]"}],"fun_headline_variants":["Real notebooks crush speedup myths: Modin 8%, Dask and Koalas 0%","First real-world Pandas API benchmark shows most alternatives fail","Kaggle notebooks: Modin helps 8%, Dask and Koalas 0%","Pandas alternatives stumble on real code, not toy tests","Largest Pandas API benchmark: prior speedups don't hold on real code"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the manual fixing, cleaning, adapting, and rescaling in Section 4.5 did not change which computations the notebooks perform or how they behave; if those edits did, the benchmark would be measuring edited code rather than real-world Pandas code.","fun_headline_variants_meta":{"raw":{"variants":["Real notebooks crush speedup myths: Modin 8%, Dask and Koalas 0%","First real-world Pandas API benchmark shows most alternatives fail","Kaggle notebooks: Modin helps 8%, Dask and Koalas 0%","Pandas alternatives stumble on real code, not toy tests","Largest Pandas API benchmark: prior speedups don't hold on real code"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000648,"raw_usage":{"total_tokens":3038,"prompt_tokens":1073,"completion_tokens":1965,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":689,"completion_tokens_details":{"reasoning_tokens":1863}},"tokens_in":689,"tokens_out":1965,"duration_ms":14054,"temperature":1.0,"reasoning_tokens":1863,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:25:43.720975+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the released notebooks with their original Kaggle inputs, run them as downloaded, and compare the set of Pandas API calls, column types, and row counts against the cleaned versions; any material difference in what executes would show PandasBench is not measuring the original real-world code.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Koalas is one of the three Pandas alternatives evaluated, with failures attributed to missing arguments and JVM memory limits."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The TPC benchmark family is the main candidate for repurposing, and the paper argues none of its variants fit the Pandas API."},{"cited_title":"2022.TPC Benchmark TM H Stan- dard Specification(3.0.1 ed.)","cited_arxiv_id":null,"evidence_quote":"The TPC-H specification is cited for uniform scaling and the old-sales refresh function, the contrasts to PandasBench's non-uniform scaling and data cleaning."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Kaggle is the source of the real-world notebooks and provides the original data needed for executability."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Pandera is the closest known way to generate valid input data but requires schemas, supporting the choice to collect notebooks with their original data."}],"review_version":1}