FlowETL: An Autonomous Example-Driven Pipeline for Data Engineering

Marcel Jaspars; Mattia Di Profio; Mingjun Zhong; Yaji Sripada

REVIEW 5 major objections 5 minor 2 cited by

FlowETL: An Autonomous Example-Driven Pipeline for Data Engineering

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FlowETL claims that a fully autonomous, example-driven ETL pipeline can infer and apply a transformation plan from a sample of the source plus a small target file, reaching data-quality scores of 0.94–1.0 and PlanEval scores of 0.85–1.0…

desk verdict A coherent ETL architecture whose headline claim isn't measured by its own metrics. read the letter →

arxiv 2507.23118 v1 pith:FI2QGI2I submitted 2025-07-30 cs.SE

Mattia Di Profio , Mingjun Zhong , Yaji Sripada , Marcel Jaspars This is my paper

classification cs.SE

keywords ETLautomationexample-driventransformationdatawranglingschemamatchinglargelanguagemodelsqualityengineeringpipeline

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FlowETL is an autonomous ETL architecture that infers a data-transformation plan from a sample of the source file and a small user-defined target file. The paper claims that from this pair alone, its Planning Engine can align schemas, generate transformation logic, and apply it to the full source, achieving PlanEval scores of 0.85–1.0 and data-quality scores of 0.94–1.0 across 14 datasets of different domains and formats. This matters because standard ETL tools require a human to design and maintain context-specific transformations. If the claim holds, an analyst could specify only what the output should look like and let the pipeline handle cleaning, standardisation, and loading.

What carries the argument

The carrying mechanism is the Planning Engine, which turns a sampled Internal Representation (IR) of the source and a target IR into a transformation plan structured as a Directed Acyclic Graph of Data Task Nodes (DTNs). Each DTN performs one transformation—missing value handling, duplicate row removal, or numerical outlier handling—and the engine evaluates all node-strategy permutations against a Data Quality Score (DQS) that averages missing-value, outlier, and duplicate ratios, then uses the LLM to infer schema matches and transformation logic.

What would settle it

Use a source where rare but legitimate values, such as duplicates or outliers, occur only in rows outside the sampled subset; run FlowETL with a sampling percentage that excludes those rows and compare the DQS computed on the sample with the DQS computed on the full output. If the full-run DQS drops below the sample DQS, or the rare values survive untransformed, the central generalisation assumption fails.

Watch

Extended reading notes

Core claim

The central claim is that a fully automated, example-driven ETL pipeline can generalize: given a representative sample of a source dataset and a concise target dataset, the Planning Engine constructs a transformation plan by inferring schemas, matching source columns to target columns with an LLM, and generating transformation instructions with a zero-shot LLM; an ETL Worker then applies that plan to the entire source file. The paper reports that LLM-driven schema matching outperforms algorithmic schema matching on the PlanEval metric, and that FlowETL's output quality is comparable to that of manually constructed Python pipelines while removing the manual development step.

Load-bearing premise

The plan inferred from a random sample of the source generalises to the entire source dataset; if the source is unevenly distributed, unseen rows may be mishandled.

Editorial extensions

If this is right

An analyst only needs to provide a target file of 5–20 rows to drive the transformation; no transformation code needs to be written by hand.
Because the plan is computed once and applied by separate workers, a single plan can be distributed across many ETL workers for parallel processing.
LLM-based schema matching yields higher PlanEval scores (0.85–1.0) than algorithmic matching (0.62–0.90), so replacing heuristic matching with LLM inference improves plan quality.
Output quality stays close to that of a manually built pipeline (DQS 0.94–1.0) while eliminating development time, though missing-value and duplicate rates sometimes remain higher than the baseline.
Within the tested sampling range (up to 50%), larger samples correlate with higher PlanEval scores, and planning time stays roughly between 65 and 81 seconds.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the sample-to-full generalisation assumption holds, new data-quality tasks such as anomaly detection on non-numerical columns can be added as new Data Task Nodes without changing the overall planning workflow—a direction the paper only gestures at.
The fixed two-LLM-call cost model implies an engineering tradeoff: under-sampling keeps costs predictable but caps the transformations the LLM can discover; a testable extension is to measure PlanEval against stratified sampling rather than just sample size.
Because the target file is small, the quality of the inferred plan likely depends on how well the target schema covers the source schema; a sparse target may leave columns unmatched, and this dependency is not experimentally separated in the paper.
The same plan-by-sample, apply-to-full pattern could be tested on streaming data, where the sample is a time window and the plan must adapt as the distribution drifts; the paper does not discuss this scenario.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Read it. The architecture is real and clearly described, and the authors are honest about limitations. But the evaluation does not measure the thing the paper claims. The headline numbers—PlanEval 0.85–1.0, DQS 0.94–1.0—don't establish that FlowETL standardises source data according to the target. PlanEval is explicitly optimistic: an operation in the ground truth that is 'not correct' still gets 0.5, hallucinated extra operations are ignored, and the score never looks at the values the compiled logic actually produces. DQS is target-agnostic: it only tracks missing/duplicate/outlier ratios, so a plan that drops problematic rows can score 1.0 while failing to produce the intended target format. The stress-test is right: no experiment compares transformed rows to ground-truth rows at cell level. So the generalisation claim rests on metrics that are insensitive to it.

What the paper does well: it integrates existing ideas—LLM schema matching (ReMatch/Magneto style), example-driven transformation inference (Foofah/TBE), and a DAG-based pipeline with Kafka and observers—into a coherent autonomous system. The v1 vs v2 comparison gives a plausible, if weak, signal that LLM-based schema matching helps. The authors explicitly acknowledge the sampling-generalisation assumption and the missing human-in-the-loop evaluation. That counts for something.

Soft spots, in order of severity: (1) the metric mismatch above; (2) the evaluation corpus is artificial—Kaggle datasets are pre-cleaned, then repolluted by a script, and each ground truth is a 5–7 row target hand-built by the authors; (3) no code or data released, no repeated runs, so no variance or reproducibility; (4) the Bonobo comparison is not a meaningful benchmark since Bonobo is manually coded and the reported time excludes development. The sampling experiment uses one dataset and fails above p=0.5, and the authors say so.

This is a plausible systems paper that deserves referee time, but the evidence base needs real strengthening before the claims can be accepted. For a reader building LLM-based data engineering tools, the architecture is worth a look. I would not cite it yet. Send to review, but the reviewers should demand cell-level accuracy metrics, standard datasets, and released artifacts.

Referee Report

5 major / 5 minor

Summary. The paper presents FlowETL, an example-driven autonomous ETL pipeline architecture that infers transformation plans from a source dataset sample and a small user-defined target dataset, and then applies the inferred plan to the full source. The system comprises Observers, a Planning Engine that combines data-quality handlers (missing values, duplicates, outliers) with LLM-based schema matching and transformation-logic inference, an ETL Worker that executes the plan, and a Reporting Engine for monitoring. The evaluation uses 14 datasets (mostly Kaggle-derived, artificially polluted) and reports high Data Quality Scores (DQS 0.94–1.0) and PlanEval scores (0.85–1.0), plus a comparison with Bonobo ETL. The central claim is that FlowETL can autonomously standardise and prepare input datasets according to a target while maintaining output quality comparable to manually built pipelines.

Significance. If the claims were fully supported, FlowETL would be a meaningful step toward reducing human effort in ETL by using example-driven LLM-based planning. The architecture is well described, and the paper makes a serious attempt at a modular design with explicit abstractions (internal representation, data task nodes, messaging-topics) and an evaluation across diverse formats. The use of LLMs for both schema matching and transformation inference is timely. However, the current evidence does not close the gap from reported scores to the headline generalisation claim: the two principal metrics, DQS and PlanEval, are each insufficient in ways the paper itself partly acknowledges, and the reported DQS numbers are internally inconsistent with the stated formula. The paper would benefit from a direct cell-level comparison against ground-truth transformed outputs and from testing the sample-to-full-dataset generalisation assumption that the architecture explicitly relies on.

major comments (5)

[§4.1 (PlanEval)] The PlanEval metric, as defined by the text and the formula, does not measure whether the final output rows match the ground-truth target rows at the cell level. It only checks whether the inferred plan operations are present in the ground-truth plan and whether they are classified as correct or incorrect, with incorrect operations still receiving 0.5 and extra (hallucinated) operations being ignored. The paper reports PlanEval scores of 0.85–1.0, but these scores do not establish that the compiled 'Logic' string executed by the ETL Worker actually produces the intended values, schema, or formatting. A plan that is operationally close to the ground truth but fails at execution could still receive a high PlanEval score. The authors should add an output-level evaluation that compares the transformed rows or cells against the ground-truth transformed dataset.
[§3.4 and Table 6] The DQS metric is target-agnostic: it averages the ratios of missing values, outliers, and duplicate rows, but it does not check whether the output conforms to the target schema or whether the cell values match the intended transformation. Consequently, a plan that drops or imputes problematic rows/columns can inflate DQS while still failing to produce the target format. More concretely, the DQS values reported in Table 6 are not reproducible from the stated formula and the percentages in the same table. For example, Amazon Stock has 0.00% missing, 3.41% duplicate, and 0.00% outlier, which gives DQS = 1 − (0 + 0.0341 + 0)/3 ≈ 0.99, not the reported 0.96; Chess Games and Smartwatch Readings have all-zero percentages, which should give DQS = 1.0, not 0.97 and 0.96. This inconsistency undermines the quantitative claims and needs to be resolved, either by correcting the numbers or by clarifying how DQS is actually computed.
[§3.2, §3.10, and §4.3] The paper explicitly assumes that a transformation plan computed on a random sample generalises to the entire source dataset (Section 3.10 and Section 3.2), yet no experiment tests this assumption. The evaluation uses p = max(object_count × 0.05, 50), meaning that for large datasets the plan is computed on only 5% of the data, and the reported PlanEval and DQS scores are not broken down by whether they were measured on the sample or the full dataset. The sampling-percentage experiment (Section 4.3) studies correlation with the sample size but does not measure whether the plan produced on the sample works correctly on the full source. The authors should explicitly compare plan correctness or output-level accuracy on the full dataset against the sample-level scores, especially for datasets where Table 6 shows unresolved missing values or outliers (e.g., Netflix Users, Pixar Films, Flight Routes).
[§4.4 (Bonobo comparison)] The comparison with Bonobo ETL does not support the claimed 'autonomy' advantage. The Bonobo pipelines were manually constructed by the authors following the ground-truth transformations, while FlowETL received pre-defined target datasets with 5–7 entries; the development time for the Bonobo workflows is explicitly excluded from the time measurements, and the paper acknowledges that no developer study was conducted. As presented, the comparison only shows that FlowETL achieves DQS values comparable to manually written pipelines, which is insufficient because DQS is not a measure of correctness against the target. The authors should either reframe the Bonobo comparison as a simple sanity check or conduct a human-in-the-loop study that measures end-to-end effort for both approaches.
[§4.1 (evaluation corpus)] The evaluation corpus is constructed from Kaggle datasets that the paper itself describes as pre-cleaned, which are then artificially polluted to contain roughly 40% missing values, 20% duplicates, and 5–10% outliers. The ground-truth targets and plans are all hand-authored by the authors. This setup is acceptable as a proof-of-concept, but it does not support the strong generalisation claim across '14 diverse datasets' in real-world conditions. At least a small-scale test on naturally messy, non-pre-cleaned data would strengthen the external validity; otherwise, the paper should temper the generalisation language and present the results as a controlled demonstration.

minor comments (5)

[§4.3] The text says 'a series of p values from 5% to %100' but later states that the pipeline failed for any p > 0.5, so the actual tested range is 5% to 50%; please correct the range description.
[§4.1 (PlanEval description)] The sentence 'rewards correct operations within a plan and ignores incorrect ones' is contradicted by the following description, in which an incorrect operation that exists in the ground truth still earns 0.5; please rephrase to avoid ambiguity.
[§4.4] The phrase 'The author first learned to use the basic functionalities' should be 'The authors first learned...' for consistency with the rest of the paper.
[§3.4] The DQS formula definition mixes ratios and percentages: the variables M, O, and D are defined as ratios (missing entries ÷ n, etc.), but Tables 5 and 6 report these as percentages; please clarify which representation is used in the formula.
[§2.4] The claim that Foofah 'successfully synthesised correct programs in 90% of test cases' lacks a direct citation to the specific evaluation details; consider referencing the Foofah paper's experimental section more explicitly.

Circularity Check

1 steps flagged · score 6.0 of 10

DQS is both the planner's optimization target and the headline quality metric, making the reported data-quality scores partially circular; PlanEval provides independent but optimistic operation-level evidence.

fitted input called prediction [Section 3.4 (DQS definition and plan objective); Section 3.8 (plan selection); Section 4.4 / Table 6 (reported DQS)]
"The Planning Engine is responsible for generating a transformation plan that, when applied to a sampled source dataset, maximises its Data Quality Score (DQS) ... 𝐷𝑄𝑆(IR) = 1−(𝑀+𝑂+𝐷)÷ 3 ... [Plan Steps -] a sequence of DTNs designed to maximise the data quality (DQS) on the sample IR. ... FlowETL achieved post-ETL data quality scores ranging from 0.94 to 1.0."

The headline evidence of output quality is the same quantity the planner is built to maximise. DQS is defined as 1 minus the average of missing-value, outlier, and duplicate ratios, and the three DTNs (MVH, DRH, NOH) are precisely the handlers that drive those three ratios to zero. The Planning Engine enumerates node-strategy combinations and selects the plan yielding the highest DQS on the sample. Reporting the resulting DQS (0.94-1.0) as an outcome is therefore reporting the optimised objective rather than an independent measure of whether the output matches the user's target.

full rationale

FlowETL's derivation chain contains one significant circular step: the Data Quality Score used as evidence is the very objective the Planning Engine maximizes. Because DQS is a linear combination of missing/outlier/duplicate rates and the DTNs are purpose-built to minimize those rates, a high post-ETL DQS is substantially forced by the planner's design. The PlanEval comparison to human-defined ground-truth plans provides independent content, but it is an optimistic operation-level metric that never inspects the transformed cell values, so it cannot fully rescue the central generalization claim from the DQS circularity. The only self-citation (reference [2] for the 80% wrangling-time statistic) is motivational and not load-bearing. The acknowledged sample-representativeness assumption is an explicit limitation, not a circularity. Overall score 6: partial circularity through the optimized DQS metric.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central claim rests on several domain assumptions about data distribution, file structure, and LLM capability. Free parameters are mostly standard or heuristic choices; no model parameters are fitted to the evaluation data.

free parameters (4)

MAD scale factor b = 1.48
Recommended by Singh et al. [31]; chosen by hand, not fitted to the evaluation data. Used in the Numerical Outliers Handler (Section 3.5.3).
MAD outlier threshold a = 3.0
Recommended by Singh et al. [31]; chosen by hand. Sets the outlier boundary in the Numerical Outliers Handler.
DQS early-stop threshold = 0.95
Plan search terminates when DQS exceeds 0.95 (Section 3.8); chosen by the authors to reduce computational cost.
Sampling percentage p = max(0.05 * objects, 50)
The Source Observer samples a subset; the value is a system parameter, and the paper shows PlanEval increases with p (Section 4.3).

assumptions (6)

domain assumption Source data within a JSON file is stored in a list of JSON objects.
Stated in Section 3.1.2; the IR extraction recursively finds the first list and treats it as the table.
domain assumption Data in the source dataset is randomly distributed, making a random sample representative.
Stated in Section 3.2; the Source Observer samples rows, and the authors note this may compromise plan quality if the data is not random.
domain assumption Target files are small (5-20 objects) and are not sampled.
Stated in Section 3.2; the design assumes the target file is small enough to process in full.
domain assumption Schema matching can be performed using only column names and types, ignoring values.
Stated in Section 3.7; the authors argue values may be altered by transformations, so matching is agnostic to values.
domain assumption A column containing exactly two distinct values is boolean.
Heuristic in Algorithm 1 (Section 3.6) to handle binary encodings; the authors note the risk of misclassification when sampling produces only two values.
domain assumption The LLM can infer correct transformation functions from a small number of examples via zero-shot prompting.
Core to the system (Section 3.9); no fine-tuning is used, and the paper relies on the LLM's prior knowledge and generalisation ability.

how reviews work

0 comments

Cite this review

Pith. "Pith review of FlowETL: An Autonomous Example-Driven Pipeline for Data Engineering." pith.science (2026). https://pith.science/paper/FI2QGI2I

@misc{pith2026250723118,
  author       = {Pith},
  title        = {Pith review of: FlowETL: An Autonomous Example-Driven Pipeline for Data Engineering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FI2QGI2I}},
  note         = {Machine review of arXiv:2507.23118}
}

read the original abstract

The Extract, Transform, Load (ETL) workflow is fundamental for populating and maintaining data warehouses and other data stores accessed by analysts for downstream tasks. A major shortcoming of modern ETL solutions is the extensive need for a human-in-the-loop, required to design and implement context-specific, and often non-generalisable transformations. While related work in the field of ETL automation shows promising progress, there is a lack of solutions capable of automatically designing and applying these transformations. We present FlowETL, a novel example-based autonomous ETL pipeline architecture designed to automatically standardise and prepare input datasets according to a concise, user-defined target dataset. FlowETL is an ecosystem of components which interact together to achieve the desired outcome. A Planning Engine uses a paired input-output datasets sample to construct a transformation plan, which is then applied by an ETL worker to the source dataset. Monitoring and logging provide observability throughout the entire pipeline. The results show promising generalisation capabilities across 14 datasets of various domains, file structures, and file sizes.

Figures

Figures reproduced from arXiv: 2507.23118 by the authors.

**Figure 2.** JSON payload with associated objects reconstruction key (top) and its internal representation within FlowETL (bottom). 1 { "from": observerName, 2 "contents": { 3 "filename": filename, 4 "objectsCount": count, 5 "filesizeMBs": fileSize 6 }} 7 (a) Runtime metrics payload 1 { 2 "name": filename, 3 "reconstructionKey": key, 4 "contents": sampledIR 5 } 6 (b) Source file artifacts payload [PITH_FULL_IMAGE:figures/full_… view at source ↗

**Figure 3.** Structure of Observer payloads sent to the Messag [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Messaging system topics overview. Blue edges de [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: PlanEval results from the comparison between the Planning Engine using algorithmic schema matching and example-based prompting (v1, blue) vs the Planning Engine using LLMs for both schema matching and transformation inference, without example-based prompting (v2, orang…

**Figure 6.** Figure 6: Comparison of the DQS achieved on each evaluation [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

**Figure 7.** Figure 7: Time elapsed (left), PlanEval score (middle), Maximum Data Quality Score (DQS) achieved (right) on the Amazon Stock dataset for varying sampling percentage p values datasets. Repeating the experiment with additional datasets, potentially capping p to ensure that the I…

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

EcoTable: Cost-effective Table Integration in Data Lakes for Natural Language Queries
cs.DB 2026-06 unverdicted novelty 6.0 of 10

EcoTable is the first NL-based data integration framework that builds a join-likelihood graph, uses two-stage schema linking and Steiner tree search to find paths, then generates transformations with LLMs, reporting >...
EcoTable: Cost-effective Table Integration in Data Lakes for Natural Language Queries
cs.DB 2026-06 conditional novelty 6.0 of 10

Query-driven table integration that uses Steiner-tree search to choose which joins LLMs must verify, reporting 30%+ accuracy gains at 5x lower LLM cost.

Reference graph

Works this paper leans on

36 extracted references · 28 canonical work pages · cited by 1 Pith paper

[1]

Data wrangling for big data: Challenges and Opportunities

T. Furche, G. Gottlob, L. Libkin, G. Orsi, and N. Paton, “Data wrangling for big data: Challenges and Opportunities”, in Proc. 19th Int. Conf. Extending Database Technology (EDBT), 2016, pp. 473–478

work page 2016
[2]

A survey of pipeline tools for data engi- neering

A. Mbata, Y. Sripada, and M. Zhong, “A survey of pipeline tools for data engi- neering”, arXiv preprint arXiv:2406.08335, 2024

arXiv 2024
[3]

Automated ETL testing on the data quality of a data warehouse

S. B. Dakrory, T. M. Mahmoud, A. A. Ali et al., “Automated ETL testing on the data quality of a data warehouse”, Int. J. Comput. Appl., vol. 131, no. 16, pp. 9–16, 2015

work page 2015
[4]

A taxonomy of ETL activities

P. Vassiliadis, A. Simitsis, and E. Baikousi, “A taxonomy of ETL activities”, in Proceedings of the ACM Twelfth International Workshop on Data Ware- housing and OLAP (DOLAP ’09), Hong Kong, China, pp. 25–32, 2009. doi: 10.1145/1651291.1651297

work page arXiv 2009
[5]

Role of Machine Learning in ETL Au- tomation

K. C. Mondal, N. Biswas, and S. Saha, “Role of Machine Learning in ETL Au- tomation”, in Proceedings of the 21st International Conference on Distributed Computing and Networking (ICDCN ’20), Kolkata, India, Article 57, 6 pp., 2020. doi: 10.1145/3369740.3372778

work page arXiv 2020
[6]

Automating data pipelines with AI: From data engineering to intelligent systems

N. Devarasetty, “Automating data pipelines with AI: From data engineering to intelligent systems”, Revista de Inteligencia Artificial en Medicina, vol. 9, no. 1, 2018

work page 2018
[7]

LLMs for data engineering on enterprise data

J.-M. Bodensohn, U. Brackmann, L. Vogel, M. Urban, A. Sanghi, and C. Binnig, “LLMs for data engineering on enterprise data”, Proc. VLDB Endow., vol. 2150, p. 8097, 2024

work page 2024
[8]

Auto-transform: Learning-to-transform by patterns

Z. Jin, Y. He, and S. Chaudhuri, “Auto-transform: Learning-to-transform by patterns”, Proc. VLDB Endow., vol. 13, no. 12, pp. 2368–2381, 2020

work page 2020

Show all 36 references

[9]

Foofah: Transforming data by example

Z. Jin, M. R. Anderson, M. Cafarella, and H. V. Jagadish, “Foofah: Transforming data by example”, in Proc. 2017 ACM Int. Conf. on Management of Data (SIGMOD ’17), Chicago, IL, USA, pp. 683–698, 2017. doi: 10.1145/3035918.3064034

2017
[10]

and Chaudhuri, S., 2023

Li, P., He, Y., Yashar, D., Cui, W., Ge, S., Zhang, H., Fainman, D.R., Zhang, D. and Chaudhuri, S., 2023. Table-gpt: Table-tuned gpt for diverse table tasks. arXiv preprint arXiv:2310.09263

2023 arXiv
[11]

and Wu, T., 2023, December

Sharma, A., Li, X., Guan, H., Sun, G., Zhang, L., Wang, L., Wu, K., Cao, L., Zhu, E., Sim, A. and Wu, T., 2023, December. Automatic data transformation using 11 Mattia Di Profio, Mingjun Zhong, Yaji Sripada, and Marcel Jaspars large language model-an experimental study on buil...

2023
[12]

A formal basis for the heuristic deter- mination of minimum cost paths

P. E. Hart, N. J. Nilsson, and B. Raphael, “A formal basis for the heuristic deter- mination of minimum cost paths”, IEEE Trans. Syst. Sci. Cybern., vol. 4, no. 2, pp. 100–107, 1968

1968
[13]

DataXFormer: An interactive data transformation tool

J. Morcos, Z. Abedjan, I. F. Ilyas, M. Ouzzani, P. Papotti, and M. Stonebraker, “DataXFormer: An interactive data transformation tool”, in Proc. 2015 ACM SIGMOD Int. Conf. on Management of Data, pp. 883–888, 2015

2015
[14]

Transform- data-by-example (TDE): An extensible search engine for data transformations

Y. He, X. Chu, K. Ganjam, Y. Zheng, V. Narasayya, and S. Chaudhuri, “Transform- data-by-example (TDE): An extensible search engine for data transformations”, Proc. VLDB Endow., vol. 11, no. 10, pp. 1165–1177, 2018

2018
[15]

Learning semantic string transformations from exam- ples

R. Singh and S. Gulwani, “Learning semantic string transformations from exam- ples”, arXiv preprint arXiv:1204.6079, 2012

2012 arXiv
[16]

Incremental schema matching

P. A. Bernstein, S. Melnik, and J. E. Churchill, “Incremental schema matching”, in Proc. VLDB, vol. 6, Seoul, Korea, pp. 1167–1170, 2006

2006
[17]

On matching schemas automatically

E. Rahm and P. A. Bernstein, “On matching schemas automatically”, VLDB J., vol. 10, no. 4, pp. 334–350, 2001

2001
[18]

Semantic schema matching

F. Giunchiglia, P. Shvaiko, and M. Yatskevich, “Semantic schema matching”, in OTM Confederated International Conferences “On the Move to Meaningful Internet Systems”, pp. 347–365, 2005

2005
[19]

Managing uncertainty in schema matching with top-k schema mappings

A. Gal, “Managing uncertainty in schema matching with top-k schema mappings”, in Journal on Data Semantics VI, pp. 90–114, Springer, 2006

2006
[20]

MaxSM: A multi-heuristic approach to XML schema matching

M. Beg, L. Charlin, and J. So, “MaxSM: A multi-heuristic approach to XML schema matching”, J. Comput. Sci. Technol., 2006

2006
[21]

ReMatch: Retrieval enhanced schema matching with LLMs

E. Sheetrit, M. Brief, M. Mishaeli, and O. Elisha, “ReMatch: Retrieval enhanced schema matching with LLMs”, arXiv preprint arXiv:2403.01567, 2024

2024 arXiv
[22]

Magneto: Combining small and large language models for schema matching

Y. Liu, E. Pena, A. Santos, E. Wu, and J. Freire, “Magneto: Combining small and large language models for schema matching”, arXiv preprint arXiv:2412.08194, 2024

2024 arXiv
[23]

Mckinney, Wes. (2011). pandas: a Foundational Python Library for Data Analysis and Statistics. Python High Performance Science Computer

2011
[24]

Xin, Patrick Wendell, Tathagata Das, Michael Armbrust, Ankur Dave, Xiangrui Meng, Josh Rosen, Shivaram Venkataraman, Michael J

Matei Zaharia, Reynold S. Xin, Patrick Wendell, Tathagata Das, Michael Armbrust, Ankur Dave, Xiangrui Meng, Josh Rosen, Shivaram Venkataraman, Michael J. Franklin, Ali Ghodsi, Joseph Gonzalez, Scott Shenker, and Ion Stoica. 2016. Apache Spark: a unified engine for big data pro...

2016 doi
[25]

https://kafka.apache.org/
[26]

Publish/Subscribe, pages 2208–2211

Hans-Arno Jacobsen. Publish/Subscribe, pages 2208–2211. Springer US, Boston, MA, 2009. ISBN 978-0-387-39940-9. doi: 10.1007/978-0-387-39940-9_1181

2009 doi
[27]

Kafka, The Definitive Guide, chapter 6, Reliable Data Delivery, pages 115–134

Gwen Shapira Todd Palino Neha Narkhede. Kafka, The Definitive Guide, chapter 6, Reliable Data Delivery, pages 115–134. O’Reilly, 2017

2017
[28]

Data engineering for data analytics: A classification of the issues, and case studies

A. Nazabal, C. K. I. Williams, G. Colavizza, C. R. Smith, and A. Williams, “Data engineering for data analytics: A classification of the issues, and case studies”, arXiv preprint arXiv:2004.12929, 2020

2004 arXiv
[29]

Next-generation etl framework to address the challenges posed by big data

Syed Muhammad Fawad Ali. Next-generation etl framework to address the challenges posed by big data. In DOLAP, 2018

2018
[30]

Outlier detection: how to threshold outlier scores?

J. Yang, S. Rahardja, and P. Fränti, “Outlier detection: how to threshold outlier scores?”, in Proc. Int. Conf. Artificial Intelligence, Information Processing and Cloud Computing (AIIPCC ’19), Sanya, China, 2019, Art. no. 37, pp. 1–6, doi: 10.1145/3371425.3371427

2019
[31]

Outlier detection: Applications and techniques

K. Singh and S. Upadhyaya, “Outlier detection: Applications and techniques”, Int. J. Comput. Sci. Issues (IJCSI), vol. 9, no. 1, pp. 307, 2012

2012
[32]

fireindark707, Python Schema Matching, GitHub repository, https://github.com/ fireindark707/Python-Schema-Matching, accessed May 27, 2025

2025
[33]

ELT-Bench: An End-to-End Benchmark for Evaluat- ing AI Agents on ELT Pipelines

T. Jin, Y. Zhu, and D. Kang, “ELT-Bench: An End-to-End Benchmark for Evaluat- ing AI Agents on ELT Pipelines”, arXiv preprint arXiv:2504.04808, 2025

2025 arXiv
[34]

College admissions and the stability of marriage

D. Gale and L. S. Shapley, “College admissions and the stability of marriage”, The American Mathematical Monthly, vol. 69, no. 1, pp. 9–15, 1962

1962
[35]

Problems with cosine as a measure of embedding similarity for high frequency words

K. Zhou, K. Ethayarajh, D. Card, and D. Jurafsky, “Problems with cosine as a measure of embedding similarity for high frequency words”, arXiv preprint arXiv:2205.05092, 2022

2022 arXiv
[36]

Can LLM already serve as a database interface? A big bench for large-scale database grounded text-to-SQLs

J. Li, B. Hui, G. Qu, J. Yang, B. Li, B. Li, B. Wang, B. Qin, R. Geng, N. Huo et al., “Can LLM already serve as a database interface? A big bench for large-scale database grounded text-to-SQLs”, Advances in Neural Information Processing Systems, vol. 36, 2024. 12

2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Data wrangling for big data: Challenges and Opportunities

T. Furche, G. Gottlob, L. Libkin, G. Orsi, and N. Paton, “Data wrangling for big data: Challenges and Opportunities”, in Proc. 19th Int. Conf. Extending Database Technology (EDBT), 2016, pp. 473–478

work page 2016

[2] [2]

A survey of pipeline tools for data engi- neering

A. Mbata, Y. Sripada, and M. Zhong, “A survey of pipeline tools for data engi- neering”, arXiv preprint arXiv:2406.08335, 2024

arXiv 2024

[3] [3]

Automated ETL testing on the data quality of a data warehouse

S. B. Dakrory, T. M. Mahmoud, A. A. Ali et al., “Automated ETL testing on the data quality of a data warehouse”, Int. J. Comput. Appl., vol. 131, no. 16, pp. 9–16, 2015

work page 2015

[4] [4]

A taxonomy of ETL activities

P. Vassiliadis, A. Simitsis, and E. Baikousi, “A taxonomy of ETL activities”, in Proceedings of the ACM Twelfth International Workshop on Data Ware- housing and OLAP (DOLAP ’09), Hong Kong, China, pp. 25–32, 2009. doi: 10.1145/1651291.1651297

work page arXiv 2009

[5] [5]

Role of Machine Learning in ETL Au- tomation

K. C. Mondal, N. Biswas, and S. Saha, “Role of Machine Learning in ETL Au- tomation”, in Proceedings of the 21st International Conference on Distributed Computing and Networking (ICDCN ’20), Kolkata, India, Article 57, 6 pp., 2020. doi: 10.1145/3369740.3372778

work page arXiv 2020

[6] [6]

Automating data pipelines with AI: From data engineering to intelligent systems

N. Devarasetty, “Automating data pipelines with AI: From data engineering to intelligent systems”, Revista de Inteligencia Artificial en Medicina, vol. 9, no. 1, 2018

work page 2018

[7] [7]

LLMs for data engineering on enterprise data

J.-M. Bodensohn, U. Brackmann, L. Vogel, M. Urban, A. Sanghi, and C. Binnig, “LLMs for data engineering on enterprise data”, Proc. VLDB Endow., vol. 2150, p. 8097, 2024

work page 2024

[8] [8]

Auto-transform: Learning-to-transform by patterns

Z. Jin, Y. He, and S. Chaudhuri, “Auto-transform: Learning-to-transform by patterns”, Proc. VLDB Endow., vol. 13, no. 12, pp. 2368–2381, 2020

work page 2020