REVIEW 2 major objections 6 minor 31 references
Text-to-SQL Oriented to the Process Mining Domain: A PT-EN Dataset for Query Translation
T0 review · 2 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read This paper introduces text-2-SQL-4-PM, the first human-curated, bilingual Portuguese-English benchmark dataset for the text-to-SQL task in the process mining domain, and shows with a GPT-3.5 Turbo baseline that the dataset supports evaluati
desk verdict Solid bilingual text-to-SQL benchmark for process mining, the first with gold SQL; the one real caveat is the undocumented 'slightly modified' BPI 2020 excerpts, which is fixable. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying object is the dataset triple: 1,655 utterances paired with 205 gold SQL statements per language and ten qualifiers. Three mechanisms do the work. First, the single non-normalized event log table — the shape any XES event log takes when loaded into a relational database — makes case-level and temporal queries require self-joins, GROUP BY, and HAVING, so the dataset's difficulty is structural, not cosmetic. Second, the ten qualifiers (event vs. case level; control-flow/temporal/resource/cost perspective; wh-question class; aggregation and HAVING presence; Spider hardness; base vs. paraphrase; value/generic/domain vocabulary) convert the collection into a diagnostic instrument for
What would settle it
Run all 205 gold SQL statements against the original, unmodified BPI Challenge 2020 event log and check whether each execution result still answers its paired natural language utterance, with independent experts re-judging any ambiguous cases. Any gold statement whose result ceases to match its utterance under the original log's semantics would show that the published dataset's answers do not transfer beyond the modified excerpts.
Extended reading notes
Core claim
text-2-SQL-4-PM is a human-curated, bilingual benchmark built from the BPI Challenge 2020 event log of a university travel-reimbursement process. Construction ran in three phases: 42 students generated 237 Portuguese utterance-SQL pairs across six business processes; three process-mining experts curated the content, adapted everything to the single travel process, verified every SQL statement by execution, and attached ten qualifiers; and a final phase added 1,450 manually written paraphrases plus professional English translations, with parallel Portuguese- and English-valued event logs so both languages can be evaluated on equal footing. The dataset's central design choice is the single, no
Load-bearing premise
The dataset stands on the correctness of its 205 gold SQL statements against the published event log data; the authors disclose that the excerpts were 'slightly modified from the original event log to facilitate a more accurate analysis of the correctness of the SQL statements,' so if those edits changed the data's semantics, the gold answers and the baseline numbers would not carry over to the original BPI Challenge 2020 log.
Editorial extensions
If this is right
- Text-to-SQL systems can now be fine-tuned and evaluated on process mining's characteristic single-table, case-oriented queries instead of only cross-domain multi-table schemas.
- The qualifiers turn evaluation into diagnosis: a system's score can be decomposed by event/case level, vocabulary kind, question type, and SQL complexity, showing where improvement effort belongs.
- The GPT-3.5 Turbo reference numbers — about 32% structural match and 45-48% execution accuracy — give later systems concrete targets to beat on this domain.
- The Portuguese branch, translated and reviewed by humans, becomes a resource for Portuguese text-to-SQL that is not a machine translation of an English benchmark.
- Because every base utterance carries several expert paraphrases, the dataset supports consistency checks: a system that handles only one phrasing of a request will be exposed by its paraphrase set.
Reading between the lines
- The baseline's failure pattern — 'domain' vocabulary queries and case-level reasoning score worst while value-anchored queries score best — points to schema linking, not SQL generation, as the binding constraint in this domain. A testable extension would augment prompts with a process-mining glossary or schema-linking exemplars and measure whether execution accuracy improves.
- The ten-qualifier annotation scheme is not tied to the travel-reimbursement process; applying the same pipeline to other BPI Challenge event logs would grow a family of comparable process-mining text-to-SQL benchmarks, letting models train across logs while keeping one evaluation schema.
- The temporal-ordering failures suggest a next evaluation axis the current indicators do not isolate: sequence-aware correctness, where a query is judged right only if it preserves the order of events within cases, not just the resulting row set.
- Because all paraphrases were hand-written by process-mining experts, an open question the paper does not address is whether expert paraphrases are systematically harder than machine-generated ones; comparing difficulty distributions could tell practitioners whether human paraphrase generation is worth the cost.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces text-2-SQL-4-PM, a bilingual (Portuguese-English) benchmark dataset for the text-to-SQL task in the process mining domain. The dataset is built around event logs from the BPI Challenge 2020 'international declaration of academic trip' process and contains 1,655 natural language utterances (205 base utterances plus 1,450 human-generated paraphrases), 205 gold SQL statements, and ten qualifiers capturing process-mining, natural-language, and SQL perspectives. The construction is described as a three-phase process: student-generated utterance/SQL pairs, expert curation and domain adaptation by three process-mining experts, and manual paraphrase creation plus professional translation. The paper reports descriptive statistics for the qualifiers and presents a baseline using GPT-3.5 Turbo with zero-shot prompts, evaluated with both a structure indicator (exact set match without values) and a run indicator (execution accuracy). The authors claim this is the first human-curated bilingual text-to-SQL benchmark specifically oriented to process mining.
Significance. If the dataset and its accompanying event-log excerpts are released in a reproducible form, this is a useful new resource. The domain is genuinely underserved: existing text-to-SQL benchmarks are mostly cross-domain and general-purpose, while process-mining-specific question datasets do not provide gold SQL. The paper's construction process is described in unusual detail, including the use of multiple business processes in generation and later adaptation to a single process, human paraphrase creation without LLM assistance, and expert validation. The qualifier scheme is thoughtful and supports fine-grained analysis. The GPT-3.5 Turbo baseline is a reasonable first check and the use of both structural and execution metrics is standard. The main risk is portability and verifiability of the gold SQL against the canonical BPI Challenge 2020 log, because the paper states that the event-log excerpts were 'slightly modified' without specifying how.
major comments (2)
- [§3.1, footnote 5] The paper states that the event log excerpts were 'slightly modified from the original event log to facilitate a more accurate analysis of the correctness of the SQL statements,' but it never specifies the modifications. The entire validity of the benchmark rests on the correctness of the gold SQL, and that correctness is asserted with respect to these modified excerpts. Without a precise record of the modifications—e.g., a transformation script, a schema diff, or a mapping between original and modified rows—external users cannot verify the gold SQL against the canonical BPI Challenge 2020 log, and the dataset's portability and reproducibility are compromised. This is a load-bearing gap for the central dataset contribution.
- [§4.2, Table 12 and surrounding text] The text claims that 'in the descriptive analysis class of Qualifier 2, English shows a significant 26.7 drop in the structure indicator for paraphrased statements compared to Portuguese.' Table 12 reports, for the corresponding 'descriptive statistics' row, 23.7% (171/721) for Portuguese paraphrases and 24.1% (174/721) for English paraphrases—an increase, not a drop. No significance test is provided anywhere in the paper. This is an internal inconsistency in a central result that the authors use to illustrate the qualifier-level analysis, and it must be corrected or removed.
minor comments (6)
- [§4.1, §4.2] The structure indicator is computed on only 1,290 of the 1,655 pairs because the 365 'no hardness' class is excluded. This is disclosed, but the paper should state the coverage explicitly in the abstract-level summary of results, since 22% of the dataset is absent from the structural metric.
- [§3.1, Qualifier 8] The 'no hardness' label is defined negatively as an inability to classify due to SQLite-specific features or commands not covered by Spider. It may be clearer to name this class 'SQLite-specific' or 'out-of-Spider-scope' so that the qualifier is not interpreted as a difficulty level.
- [§4.2, Table 13] There is a typo in the 'which' row: '25.0 p8{32' is missing a closing parenthesis, probably 'p8{32q'. Please check the table formatting.
- [§5] The conclusion refers to 'PMtext2sql' while the rest of the paper uses 'text 2SQL4PM' or 'text-2-SQL-4-PM'. Unify the dataset name and the hyphen/spacing convention throughout the manuscript.
- [§4.2, Tables 19-21] In Table 21, 'for both languagems' should be 'for both languages'. Also consider adding a short explanation of why the gold SQL in Table 20 uses a self-join and timestamp comparison; the point is clear but a one-sentence gloss would help readers unfamiliar with process-mining ordering constraints.
- [§3.1] The paper does not report inter-annotator agreement or any reliability measure for the qualifier assignments. Given that the qualifiers are used for subsequent analysis, a simple agreement statistic would strengthen the claims about their meaningfulness.
Circularity Check
No significant circularity: the dataset and baseline are external artifacts; no prediction reduces to its inputs.
full rationale
text-2-SQL-4-PM is a newly constructed benchmark; its gold SQL statements are human-curated from student-generated pairs and expert review, not derived from the GPT-3.5 Turbo baseline. The baseline (Section 4) is a zero-shot application of an external LLM evaluated against those gold statements using standard structure and execution metrics; no parameter is fitted to the dataset and then renamed as a prediction. The qualifiers are annotation labels applied independently of the baseline and used only for post-hoc stratified analysis, so they do not define or force the measured performance. No load-bearing self-citation or imported uniqueness theorem appears: citations to prior work by the same group (e.g., [22]) are used only as one source of business-process descriptions in Phase 1, and no central claim depends on an unverified self-citation. The 'slightly modified' event log excerpts (Section 3.1) are a reproducibility/portability caveat, not circularity, because the benchmark is self-contained with its own provided excerpts and SQL statements. Accordingly, the circularity score is 0.
Assumptions & free parameters
assumptions (3)
- domain assumption Event logs can be represented as a single non-normalized table for SQL querying.
- ad hoc to paper The slightly modified event log excerpts preserve the semantics of the original BPI Challenge 2020 log.
- domain assumption Human expert curation ensures correct SQL labels and meaningful utterances.
Cite this review
Pith. "Pith review of Text-to-SQL Oriented to the Process Mining Domain: A PT-EN Dataset for Query Translation." pith.science (2026). https://pith.science/paper/URF33DCE
@misc{pith2026250909684,
author = {Pith},
title = {Pith review of: Text-to-SQL Oriented to the Process Mining Domain: A PT-EN Dataset for Query Translation},
year = {2026},
howpublished = {\url{https://pith.science/paper/URF33DCE}},
note = {Machine review of arXiv:2509.09684}
}
read the original abstract
This paper introduces text-2-SQL-4-PM, a bilingual (Portuguese-English) benchmark dataset designed for the text-to-SQL task in the process mining domain. Text-to-SQL conversion facilitates natural language querying of databases, increasing accessibility for users without SQL expertise and productivity for those that are experts. The text-2-SQL-4-PM dataset is customized to address the unique challenges of process mining, including specialized vocabularies and single-table relational structures derived from event logs. The dataset comprises 1,655 natural language utterances, including human-generated paraphrases, 205 SQL statements, and ten qualifiers. Methods include manual curation by experts, professional translations, and a detailed annotation process to enable nuanced analyses of task complexity. Additionally, a baseline study using GPT-3.5 Turbo demonstrates the feasibility and utility of the dataset for text-to-SQL applications. The results show that text-2-SQL-4-PM supports evaluation of text-to-SQL implementations, offering broader applicability for semantic parsing and other natural language processing tasks.
Reference graph
Works this paper leans on
-
[1]
Yu, T., Li, Z., Zhang, Z., Zhang, R. & Radev, D. Walker, M., Ji, H. & Stent, A. (eds) TypeSQL: Knowledge-Based Type-Aware Neural Text-to-SQL Genera- tion. (eds Walker, M., Ji, H. & Stent, A.) Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Paper...
work page 2018
-
[2]
Visperas, M. et al. Lee, J. H. & Shin, S. Y. (eds) On Modern Text-to-SQL Semantic Parsing Methodologies for Natural Language Interface to Databases: A Comparative Study . (eds Lee, J. H. & Shin, S. Y.) International Conference on Artificial Intelligence in Information and Communication (ICAIIC) , 390–396 (2023)
work page 2023
-
[3]
Katsogiannis-Meimarakis, G. & Koutrika, G. A survey on deep learning approaches for text-to-sql. The VLDB Journal 32, 905–936 (2023)
work page 2023
-
[4]
Yu, T. et al. Riloff, E., Chiang, D., Hockenmaier, J. & Tsujii, J. (eds) Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Seman- tic Parsing and Text-to-SQL Task . (eds Riloff, E., Chiang, D., Hockenmaier, J. & Tsujii, J.) Proceedings of the 2018 Conference on Empirical Methods in Natu- ral Language Processing, 3911–3921 (Associati...
work page 2018
-
[5]
Zhong, V., Xiong, C. & Socher, R. Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning. CoRR abs/1709.00103 (2017)
arXiv 2017
-
[6]
Li, J. et al. Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs (2023). 2305.03111
arXiv 2023
-
[7]
van der Aalst, W. M. P. Process Mining: Data Science in Action 2 edn (Springer, Heidelberg, 2016)
work page 2016
-
[8]
van der Aalst, W. M. P. & Carmona, J. (eds) Process Mining Handbook Vol. 448 of Lecture Notes in Business Information Processing (Springer, 2022). URL https://doi.org/10.1007/978-3-031-08848-3
Show all 31 references
-
[9]
Ieee standard for extensible event stream (xes) for achieving interoperabil- ity in event logs and event streams
IEEE. Ieee standard for extensible event stream (xes) for achieving interoperabil- ity in event logs and event streams. IEEE Std 1849-2016 1–50 (2016)
2016
-
[10]
& Navathe, S
Elmasri, R. & Navathe, S. Fundamentals of Database Systems (Pearson Education, 2010). 31
2010
-
[11]
& Richardson, M
Wang, B., Shin, R., Liu, X., Polozov, O. & Richardson, M. Jurafsky, D., Chai, J., Schluter, N. & Tetreault, J. (eds) RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers . (eds Jurafsky, D., Chai, J., Schluter, N. & Tetreault, J.) Proceedings of the 58th...
2020
-
[12]
& Berant, J
Bogin, B., Gardner, M. & Berant, J. Korhonen, A., Traum, D. & M` arquez, L. (eds) Representing Schema Structure with Graph Neural Networks for Text-to- SQL Parsing . (eds Korhonen, A., Traum, D. & M` arquez, L.) Proceedings of the 57th Annual Meeting of the Association for Com...
2019
-
[13]
& Chen, H
Li, H., Zhang, J., Li, C. & Chen, H. RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL (2023)
2023
-
[14]
Gao, D. et al. Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation. CoRR abs/2308.15363 (2023)
2023 arXiv
-
[15]
& Rafiei, D
Pourreza, M. & Rafiei, D. DIN-SQL: Decomposed In-Context Learning of Text-to- SQL with Self-Correction (2023). URL https://arxiv.org/abs/2304.11015. 2304. 11015
2023 arXiv
-
[16]
Dahl, D. A. et al. Allan, J. (ed.) Expanding the Scope of the ATIS Task: The ATIS-3 Corpus . (ed.Allan, J.) Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994 (1994). URL https: //aclanthology.org/H94-1010
1994
-
[17]
Zelle, J. M. & Mooney, R. J. Clancey, W. J. & Weld, D. (eds) Learning to parse database queries using inductive logic programming . (eds Clancey, W. J. & Weld, D.) Proceedings of the Thirteenth National Conference on Artificial Intelligence - Volume 2 , AAAI’96, 1050–1055 (AAA...
1996
-
[18]
& Zettlemoyer, L
Iyer, S., Konstas, I., Cheung, A., Krishnamurthy, J. & Zettlemoyer, L. Barzilay, R. & Kan, M.-Y. (eds) Learning a Neural Semantic Parser from User Feedback . (eds Barzilay, R. & Kan, M.-Y.) Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics...
2017
-
[19]
& Dillig, T
Yaghmazadeh, N., Wang, Y., Dillig, I. & Dillig, T. SQLizer: query synthesis from natural language. Proc. ACM Program. Lang. 1 (2017). URL https://doi.org/ 10.1145/3133887
2017 doi
-
[20]
Jos´ e, M. A. & Cozman, F. G. Brito, A. & Delgado, K. V. (eds)mRAT-SQL+GAP: A portuguese text-to-sql transformer . (eds Brito, A. & Delgado, K. V.) Anais da 32 X Brazilian Conference on Intelligent Systems (SBC, Porto Alegre, RS, Brasil, 2021)
2021
-
[21]
Barbieri, L., Madeira, E. R. M., Stroeh, K. & van der Aalst, W. M. P. Munoz- Gama, J. & Lu, X. (eds) Towards a Natural Language Conversational Interface for Process Mining. (eds Munoz-Gama, J. & Lu, X.) Process Mining Workshops, 268–280 (Springer International Publishing, Cham, 2022)
2022
-
[22]
Amaral, C. A. L., Fantinato, M., Reijers, H. A. & Peres, S. M. Ziemba, E. (ed.) Enhancing completion time prediction through attribute selection. (ed.Ziemba, E.) Information Technology for Management: Emerging Research and Applications , 3–23 (Springer International Publishing...
2019
-
[23]
Bpi challenge 2013, incidents (2013)
Steeman, W. Bpi challenge 2013, incidents (2013). URL https://data.4tu.nl/ articles/ /12693914/1
2013
-
[24]
Unger, A. J. et al. Maranh˜ ao, J. (ed.)Process mining-enabled jurimetrics: anal- ysis of a brazilian court’s judicial performance in the business law processing . (ed.Maranh˜ ao, J.)Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law , IC...
2021
-
[25]
Bpi challenge 2017 (2017)
van Dongen, B. Bpi challenge 2017 (2017). URL https://data.4tu.nl/articles/ / 12696884/1
2017
-
[26]
van Dongen, B. B. Bpi challenge 2015 (2015). URL https://data.4tu.nl/ collections/ /5065424/1
2015
-
[27]
BPI Challenge 2020 (2020)
van Dongen, B. BPI Challenge 2020 (2020). URL https://data.4tu.nl/collections/ /5065541/1
2020
-
[28]
& Klein, D
Zhong, R., Yu, T. & Klein, D. Webber, B., Cohn, T., He, Y. & Liu, Y. (eds) Semantic Evaluation for Text-to-SQL with Distilled Test Suites . (eds Webber, B., Cohn, T., He, Y. & Liu, Y.) Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNL...
2020
-
[29]
Dong, X. et al. C3: Zero-shot Text-to-SQL with ChatGPT (2023). 2307.07306
2023 arXiv
-
[30]
Liu, P. et al. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys 55 (2023)
2023
-
[31]
Vogelgesang, T. et al. Celonis PQL: A Query Language for Process Mining , 377– 408 (Springer International Publishing, Cham, 2022). URL https://doi.org/10. 1007/978-3-030-92875-9 13. 33
2022
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.