{"id":"7db41da3-e696-45eb-828d-4540ffbaad53","arxiv_id":"2509.09684","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"text-2-SQL-4-PM is a bilingual benchmark with 1,655 human utterances and 205 gold SQL statements for text-to-SQL in process mining.","lead":"The authors release text-2-SQL-4-PM, a Portuguese-English dataset of 1,655 natural language questions and 205 gold SQL statements for text-to-SQL in the process mining domain. It is the first human-curated benchmark with SQL labels for this domain, evaluated with a GPT-3.5 Turbo baseline.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Dataset portability hinges on undocumented 'slight' changes to the BPI 2020 event log; gold SQL correctness for the original data is unverifiable.","rationale":"The reader's conditional verdict is motivated by the same load-bearing concern I identified: the 'slightly modified' event-log excerpts are not described precisely, so the correctness and portability of the gold SQL cannot be verified. I agree that this is the weakest point in the argument. The dataset is a resource whose central value is serving as a benchmark; if the underlying data is not a faithful, documented version of the original BPI 2020 log, the benchmark's external validity is at risk. My reading does not change the verdict — CONDITIONAL remains appropriate — because the concern is concrete but potentially addressable via a public release with full documentation. I also considered whether the small number of distinct intents (205) or the single baselines (GPT-3.5 only) were more problematic, but these are clearly acknowledged limitations and not threats to the central claim's integrity. The event-log modification is the only place where the paper admits to altering the ground truth source without transparency. A precise, executable check—diffing the released logs against the original and testing all gold SQL statements on the original data—would settle whether the concern lands. I would not reject the paper outright; the issue is verifiability rather than demonstrable error.","tokens_in":21383,"tokens_out":3479,"duration_ms":35700,"concrete_test":"Download the released text-2-SQL-4-PM repository and the original BPI Challenge 2020 'international declaration of academic trip' log. Systematically diff the released event log excerpts against the original log: enumerate all row additions/deletions/order changes, value changes (timestamps, resources, costs, activity names), and any schema changes. Then execute the 205 gold SQL statements (both Portuguese and English versions) against the original unmodified log and check whether each produces the result intended by the accompanying natural-language utterance. If the gold SQL fails on the original log, require a detailed transformation specification; if it succeeds, the concern is empirically resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that text-2-SQL-4-PM is the first human-curated bilingual text-to-SQL benchmark for process mining, whose value depends on the correctness and reusability of its gold SQL statements. Section 3.1 states that the event log excerpts were 'slightly modified from the original event log to facilitate a more accurate analysis of the correctness of the SQL statements,' but it does not specify what was modified, why, or how. Without a precise record of the modifications, the benchmark's portability to the well-known BPI Challenge 2020 log is suspect: users expecting to run the provided SQL against the original (unmodified) event log may get different results, or the SQL may be tailored to a distorted version of the domain. The paper itself emphasizes the process mining context and the BPI 2020 provenance, so the modification is not a cosmetic detail but a potentially load-bearing gap. If the modifications change schema, activity labels, timestamps, or case identifiers, the dataset's utility as a shared benchmark is undermined because external evaluation cannot be reproduced on the original data. The paper does note that the dataset is self-contained (excerpts are provided in the repository), which mitigates internal inconsistency, but the lack of documented transformation rules means the claim 'adapted to the process mining domain' cannot be externally verified against the canonical source.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces text-2-SQL-4-PM, a bilingual (Portuguese-English) benchmark dataset for the text-to-SQL task in the process mining domain. The dataset is built around event logs from the BPI Challenge 2020 'international declaration of academic trip' process and contains 1,655 natural language utterances (205 base utterances plus 1,450 human-generated paraphrases), 205 gold SQL statements, and ten qualifiers capturing process-mining, natural-language, and SQL perspectives. The construction is described as a three-phase process: student-generated utterance/SQL pairs, expert curation and domain adaptation by three process-mining experts, and manual paraphrase creation plus professional translation. The paper reports descriptive statistics for the qualifiers and presents a baseline using GPT-3.5 Turbo with zero-shot prompts, evaluated with both a structure indicator (exact set match without values) and a run indicator (execution accuracy). The authors claim this is the first human-curated bilingual text-to-SQL benchmark specifically oriented to process mining.","tokens_in":21694,"tokens_out":4771,"duration_ms":60338,"significance":"If the dataset and its accompanying event-log excerpts are released in a reproducible form, this is a useful new resource. The domain is genuinely underserved: existing text-to-SQL benchmarks are mostly cross-domain and general-purpose, while process-mining-specific question datasets do not provide gold SQL. The paper's construction process is described in unusual detail, including the use of multiple business processes in generation and later adaptation to a single process, human paraphrase creation without LLM assistance, and expert validation. The qualifier scheme is thoughtful and supports fine-grained analysis. The GPT-3.5 Turbo baseline is a reasonable first check and the use of both structural and execution metrics is standard. The main risk is portability and verifiability of the gold SQL against the canonical BPI Challenge 2020 log, because the paper states that the event-log excerpts were 'slightly modified' without specifying how.","major_comments":[{"comment":"The paper states that the event log excerpts were 'slightly modified from the original event log to facilitate a more accurate analysis of the correctness of the SQL statements,' but it never specifies the modifications. The entire validity of the benchmark rests on the correctness of the gold SQL, and that correctness is asserted with respect to these modified excerpts. Without a precise record of the modifications—e.g., a transformation script, a schema diff, or a mapping between original and modified rows—external users cannot verify the gold SQL against the canonical BPI Challenge 2020 log, and the dataset's portability and reproducibility are compromised. This is a load-bearing gap for the central dataset contribution.","section":"§3.1, footnote 5"},{"comment":"The text claims that 'in the descriptive analysis class of Qualifier 2, English shows a significant 26.7 drop in the structure indicator for paraphrased statements compared to Portuguese.' Table 12 reports, for the corresponding 'descriptive statistics' row, 23.7% (171/721) for Portuguese paraphrases and 24.1% (174/721) for English paraphrases—an increase, not a drop. No significance test is provided anywhere in the paper. This is an internal inconsistency in a central result that the authors use to illustrate the qualifier-level analysis, and it must be corrected or removed.","section":"§4.2, Table 12 and surrounding text"}],"minor_comments":[{"comment":"The structure indicator is computed on only 1,290 of the 1,655 pairs because the 365 'no hardness' class is excluded. This is disclosed, but the paper should state the coverage explicitly in the abstract-level summary of results, since 22% of the dataset is absent from the structural metric.","section":"§4.1, §4.2"},{"comment":"The 'no hardness' label is defined negatively as an inability to classify due to SQLite-specific features or commands not covered by Spider. It may be clearer to name this class 'SQLite-specific' or 'out-of-Spider-scope' so that the qualifier is not interpreted as a difficulty level.","section":"§3.1, Qualifier 8"},{"comment":"There is a typo in the 'which' row: '25.0 p8{32' is missing a closing parenthesis, probably 'p8{32q'. Please check the table formatting.","section":"§4.2, Table 13"},{"comment":"The conclusion refers to 'PMtext2sql' while the rest of the paper uses 'text 2SQL4PM' or 'text-2-SQL-4-PM'. Unify the dataset name and the hyphen/spacing convention throughout the manuscript.","section":"§5"},{"comment":"In Table 21, 'for both languagems' should be 'for both languages'. Also consider adding a short explanation of why the gold SQL in Table 20 uses a self-join and timestamp comparison; the point is clear but a one-sentence gloss would help readers unfamiliar with process-mining ordering constraints.","section":"§4.2, Tables 19-21"},{"comment":"The paper does not report inter-annotator agreement or any reliability measure for the qualifier assignments. Given that the qualifiers are used for subsequent analysis, a simple agreement statistic would strengthen the claims about their meaningfulness.","section":"§3.1"}],"recommendation":"major_revision","confidential_remarks":"This is a dataset-resource paper with a sound overall structure, but the undocumented event-log modification is a correctness risk for the benchmark's central value. The requested changes—documenting or scripting the transformations and fixing the inconsistent Table 12 passage—are within the scope of a revision. I would not require algorithmic novelty, but I would ask the editor to ensure the repository is actually inspected and the transformation details are provided during revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a solid, honest dataset paper. It delivers the first human-curated bilingual (Portuguese-English) text-to-SQL benchmark for process mining with gold SQL statements—Barbieri et al. (2022) had process-mining questions but no gold SQL, so the dataset fills a real gap. The central claim holds.\n\nThe construction is the strength. Three phases (student generation, curation by three domain experts, then manual paraphrase and professional translation) are described concretely, and the ten qualifiers provide a useful taxonomy—process-mining perspective, natural language form, SQL hardness—for slicing evaluation results. The GPT-3.5 Turbo baseline uses standard metrics, exact set match without values and execution accuracy, and the failure analysis (temporal ordering, DISTINCT, implicit case-level concepts, domain vocabulary) is more informative than the aggregate numbers. The paper also states its own limits plainly in the conclusion: no discovery, conformance, or monitoring; SQLite-specific functions; single-table schema. That is the right posture for a dataset paper.\n\nThe caveat the reader flagged is real: Section 3.1 says the event log excerpts were 'slightly modified from the original event log' without specifying what changed. For a dataset with BPI 2020 provenance, that is under-documented and weakens the portability claim—you cannot verify the gold SQL against the canonical log. I rate this a moderate flaw, not a fatal one, because the dataset ships with its own excerpts and is self-consistent as a benchmark. The revision should document the modifications exactly and provide a transformation script.\n\nMinor issues: 205 gold SQL statements is small, some qualifier classes have very few instances (two 'when' utterances, eight 'who'), the GPT-3.5 baseline is dated, and there are small copyedit slips ('events log' vs 'event log'). None of these undermine the contribution. The citation pattern is fine—Spider, WikiSQL, BIRD, Barbieri, and the BPI Challenge sources are all apt.\n\nThis is for researchers at the intersection of process mining and semantic parsing. It is the only resource pairing natural language utterances with gold SQL in that domain, and the bilingual design extends its reach. Send it to a serious referee; the main instruction to the authors should be to document the event log modifications and verify the repository.","headline":"Solid bilingual text-to-SQL benchmark for process mining, the first with gold SQL; the one real caveat is the undocumented 'slightly modified' BPI 2020 excerpts, which is fixable.","tokens_in":22139,"tokens_out":5735,"would_cite":true,"duration_ms":65642,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces text-2-SQL-4-PM, the first human-curated, bilingual Portuguese-English benchmark dataset for the text-to-SQL task in the process mining domain, and shows with a GPT-3.5 Turbo baseline that the dataset supports evaluati","keywords":["text-to-SQL","process mining","benchmark dataset","bilingual","event log","semantic parsing","large language models","Portuguese"],"falsifier":"Run all 205 gold SQL statements against the original, unmodified BPI Challenge 2020 event log and check whether each execution result still answers its paired natural language utterance, with independent experts re-judging any ambiguous cases. Any gold statement whose result ceases to match its utterance under the original log's semantics would show that the published dataset's answers do not transfer beyond the modified excerpts.","tokens_in":21316,"feed_emoji":"🗄️","tokens_out":11242,"duration_ms":117753,"temperature":0.7,"pith_summary":"This paper introduces text-2-SQL-4-PM, a bilingual Portuguese-English benchmark dataset for the text-to-SQL task in the process mining domain — the first such dataset, the authors argue, to pair natural language utterances with gold SQL statements and rich annotations in this specialized field. The dataset contains 1,655 human-written utterances (205 originals and 1,450 paraphrases), 205 gold SQL statements per language, and ten qualifiers that label each query by process-mining perspective, question type, vocabulary kind, and SQL complexity, all targeting the single non-normalized table that an event log becomes in a relational database. A zero-shot GPT-3.5 Turbo baseline answers correctly about a third of the time under strict structural match and 44.5% (Portuguese) / 47.6% (English) when judged by execution results, showing both that the task is feasible and that process mining's case-level and temporal reasoning makes it genuinely hard. A sympathetic reader would take the paper's core contention to be that generic cross-domain text-to-SQL benchmarks miss exactly this difficulty, and that a dedicated, human-curated resource is needed to develop and fairly evaluate systems for it.","feed_headline":"Bilingual benchmark brings text-to-SQL to process mining","feed_subtitle":"A human-curated PT-EN set of 1,655 queries and 205 gold SQL statements gives process analysts a plain-language route to event logs.","key_machinery":"The carrying object is the dataset triple: 1,655 utterances paired with 205 gold SQL statements per language and ten qualifiers. Three mechanisms do the work. First, the single non-normalized event log table — the shape any XES event log takes when loaded into a relational database — makes case-level and temporal queries require self-joins, GROUP BY, and HAVING, so the dataset's difficulty is structural, not cosmetic. Second, the ten qualifiers (event vs. case level; control-flow/temporal/resource/cost perspective; wh-question class; aggregation and HAVING presence; Spider hardness; base vs. paraphrase; value/generic/domain vocabulary) convert the collection into a diagnostic instrument for","core_discovery":"text-2-SQL-4-PM is a human-curated, bilingual benchmark built from the BPI Challenge 2020 event log of a university travel-reimbursement process. Construction ran in three phases: 42 students generated 237 Portuguese utterance-SQL pairs across six business processes; three process-mining experts curated the content, adapted everything to the single travel process, verified every SQL statement by execution, and attached ten qualifiers; and a final phase added 1,450 manually written paraphrases plus professional English translations, with parallel Portuguese- and English-valued event logs so both languages can be evaluated on equal footing. The dataset's central design choice is the single, no","pith_inferences":["The baseline's failure pattern — 'domain' vocabulary queries and case-level reasoning score worst while value-anchored queries score best — points to schema linking, not SQL generation, as the binding constraint in this domain. A testable extension would augment prompts with a process-mining glossary or schema-linking exemplars and measure whether execution accuracy improves.","The ten-qualifier annotation scheme is not tied to the travel-reimbursement process; applying the same pipeline to other BPI Challenge event logs would grow a family of comparable process-mining text-to-SQL benchmarks, letting models train across logs while keeping one evaluation schema.","The temporal-ordering failures suggest a next evaluation axis the current indicators do not isolate: sequence-aware correctness, where a query is judged right only if it preserves the order of events within cases, not just the resulting row set.","Because all paraphrases were hand-written by process-mining experts, an open question the paper does not address is whether expert paraphrases are systematically harder than machine-generated ones; comparing difficulty distributions could tell practitioners whether human paraphrase generation is worth the cost."],"forward_implications":["Text-to-SQL systems can now be fine-tuned and evaluated on process mining's characteristic single-table, case-oriented queries instead of only cross-domain multi-table schemas.","The qualifiers turn evaluation into diagnosis: a system's score can be decomposed by event/case level, vocabulary kind, question type, and SQL complexity, showing where improvement effort belongs.","The GPT-3.5 Turbo reference numbers — about 32% structural match and 45-48% execution accuracy — give later systems concrete targets to beat on this domain.","The Portuguese branch, translated and reviewed by humans, becomes a resource for Portuguese text-to-SQL that is not a machine translation of an English benchmark.","Because every base utterance carries several expert paraphrases, the dataset supports consistency checks: a system that handles only one phrasing of a request will be exposed by its paraphrase set."],"supporting_citations":[{"why":"van Dongen, BPI Challenge 2020 — supplies the source event log whose (slightly modified) excerpts the dataset's utterances and SQL statements target.","marker":"[27]"},{"why":"Yu et al., Spider — provides the hardness criteria behind Qualifier 8 and the cross-domain benchmark standard the dataset positions itself against.","marker":"[4]"},{"why":"Zhong et al. — defines the exact-set-match-without-values and execution-accuracy indicators that form the baseline evaluation.","marker":"[28]"},{"why":"José and Cozman, mRAT-SQL+GAP — the only other Portuguese text-to-SQL dataset; the comparison point for the bilingual contribution.","marker":"[20]"},{"why":"Barbieri et al. — the prior process-mining question dataset that lacks gold SQL outputs, which text-2-SQL-4PM extends into supervised territory.","marker":"[21]"},{"why":"Dong et al., C3 — documents zero-shot ChatGPT text-to-SQL performance and supports the choice of GPT-3.5 Turbo for the baseline.","marker":"[29]"},{"why":"Li et al., BIRD — the large-scale database-grounded benchmark whose cross-domain, multi-table design motivates the need for a single-table domain-specific resource.","marker":"[6]"}],"fun_headline_variants":["PT-EN dataset turns plain text into SQL for event logs","Bilingual benchmark: natural language to SQL for process mining","Speak to event logs: 1,655-query bilingual text-to-SQL set","Human-curated bilingual dataset for querying process logs via SQL"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The dataset stands on the correctness of its 205 gold SQL statements against the published event log data; the authors disclose that the excerpts were 'slightly modified from the original event log to facilitate a more accurate analysis of the correctness of the SQL statements,' so if those edits changed the data's semantics, the gold answers and the baseline numbers would not carry over to the original BPI Challenge 2020 log.","fun_headline_variants_meta":{"raw":{"variants":["PT-EN dataset turns plain text into SQL for event logs","Bilingual benchmark: natural language to SQL for process mining","Speak to event logs: 1,655-query bilingual text-to-SQL set","Human-curated bilingual dataset for querying process logs via SQL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000798,"raw_usage":{"total_tokens":3336,"prompt_tokens":722,"completion_tokens":2614,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":466,"completion_tokens_details":{"reasoning_tokens":2538}},"tokens_in":466,"tokens_out":2614,"duration_ms":20866,"temperature":1.0,"reasoning_tokens":2538,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T19:24:30.336173+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run all 205 gold SQL statements against the original, unmodified BPI Challenge 2020 event log and check whether each execution result still answers its paired natural language utterance, with independent experts re-judging any ambiguous cases. Any gold statement whose result ceases to match its utterance under the original log's semantics would show that the published dataset's answers do not transfer beyond the modified excerpts.","supporting_citations":[{"cited_title":"BPI Challenge 2020 (2020)","cited_arxiv_id":null,"evidence_quote":"van Dongen, BPI Challenge 2020 — supplies the source event log whose (slightly modified) excerpts the dataset's utterances and SQL statements target."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Yu et al., Spider — provides the hardness criteria behind Qualifier 8 and the cross-domain benchmark standard the dataset positions itself against."},{"cited_title":"& Klein, D","cited_arxiv_id":null,"evidence_quote":"Zhong et al. — defines the exact-set-match-without-values and execution-accuracy indicators that form the baseline evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"José and Cozman, mRAT-SQL+GAP — the only other Portuguese text-to-SQL dataset; the comparison point for the bilingual contribution."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Barbieri et al. — the prior process-mining question dataset that lacks gold SQL outputs, which text-2-SQL-4PM extends into supervised territory."}],"review_version":1}