{"id":"17c5ebf9-be1d-4813-be79-ac239fc4472e","arxiv_id":"2507.08362","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A BPMN extraction pipeline using BERT, RoBERTa, CRF, and CatBoost is evaluated on an augmented PET dataset, with 15 new documents adding 32 AND gateways to improve parallel-structure detection.","lead":"This paper automates the conversion of business process text into BPMN diagrams using BERT-based entity recognition, CatBoost relation classification, and graph layout. It also contributes 15 new annotated documents with 32 parallel gateway examples, aimed at a known weak spot in process extraction: detecting parallel activities.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed AND-gateway improvement rests on an uncontrolled comparison over the authors' own tiny gold standard; 0% to 23% may reflect annotation style and split noise rather than a generalizable gain.","rationale":"I read the paper in good faith and take the reported numbers at face value. The paper is a small empirical contribution and is reasonably transparent about limitations, but the central added value—that LESCHNEIDER improves parallel-structure detection—is not yet supported by a controlled comparison. The reader's weakest assumption is the validity of the self-authored LESCHNEIDER gold standard; I agree that this is the fragile point. My reading adds a concrete internal-validity problem: the 0% to 23% improvement in Table 3 is not an ablation on a fixed test set, because the baseline is PET-CV on PET while the improved model is combined-CV on combined data. The only PET-only model tested on LESCHNEIDER (CRF Model 2) has B-AND F1=0, which shows that PET alone does not transfer to LESCHNEIDER style, but the paper does not report the symmetric condition (PET+LESCHNEIDER training, LESCHNEIDER-only test) under the same folds. Given the texts were authored by the same team and use explicit markers like 'and simultaneously', intra-corpus improvement could reflect style matching rather than robust concurrency detection. The tiny positive class and absence of significance testing make the absolute gain even less interpretable. I found no reason to question the authors' honesty; the issue is that the reported comparison is not yet controlled enough to establish the central claim. I therefore keep the CONDITIONAL verdict, with conditions strengthened to require a paired ablation and an external, independently annotated test set, not merely dataset release. I also note minor inconsistencies (abstract says 32 AND gateways while Table 2 lists 40 AND mentions, and the six pipeline test documents are not sourced) that should be clarified in revision.","tokens_in":10208,"tokens_out":8845,"duration_ms":106627,"concrete_test":"Release the LESCHNEIDER dataset and code (the manuscript mentions a GitHub repository but gives no identifier). Then run a paired ablation with identical document-level 5-fold splits of the combined corpus: in each fold, train CRF and BERT-base-cased on PET-only and on PET+LESCHNEIDER (train folds), and evaluate both on the same held-out fold. Report B-AND and I-AND F1 separately for PET-held-out and LESCHNEIDER-held-out documents, with per-fold scores and 95% bootstrap confidence intervals. Additionally, annotate with two independent annotators (reporting inter-annotator agreement) a set of 20–30 external process descriptions not authored by the team, selected from real business documentation or from an existing benchmark, and compare PET-only vs PET+LESCHNEIDER on those external AND mentions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central RQ2 claim in §5.2 rests on Table 3, but the headline rise in B-AND F1 from 0.00 to 0.23 crosses different test sets: the 0.00 is the PET-only 5-fold CV on PET, while the 0.23 is the combined PET+LESCHNEIDER 5-fold CV on the combined corpus. CRF Model 2, the only condition that isolates LESCHNEIDER as a test set, scores 0.00 on B-AND even when trained on all of PET; the paper does not report the symmetric condition needed for a fair ablation, namely a PET-only model and a PET+LESCHNEIDER model evaluated on the identical held-out folds. Because the LESCHNEIDER texts were written and annotated by the same authors, with parallelism introduced through explicit markers such as 'and simultaneously' (§3.2), a model trained on part of LESCHNEIDER can match the style of the remaining part without learning a generalizable notion of concurrency. The positive class is tiny (8 PET vs 40 LESCHNEIDER AND mentions in Tables 1 and 2; the abstract says 32 gateways), so an F1 of 23% is well within sampling noise; no per-fold scores, confidence intervals, or significance tests are reported. The pipeline F1s in §5.4 are computed on six test documents whose provenance and holding-out are not stated. The central claim therefore rests on an uncontrolled and likely optimistic comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an automated pipeline for generating BPMN models from text, combining preprocessing, NER (CRF and fine-tuned BERT/RoBERTa), relation extraction with CatBoost, entity resolution, and diagram generation. Its main contribution is the LESCHNEIDER dataset, 15 manually authored and annotated documents adding 32 AND gateways to the PET dataset, intended to improve detection of parallel structures. Three experiments address whether PET is sufficiently diverse, whether the augmented corpus improves AND-gateway detection, and whether BERT or RoBERTa is more effective. Reported results include a weighted-average CRF F1 of 0.72, a B-AND F1 improvement from 0.00 to 0.23 after adding LESCHNEIDER, BERT-base-cased as the best NER model, and a full-pipeline element F1 of 89.2% with relation F1 of 73.7%.","tokens_in":10531,"tokens_out":6254,"duration_ms":66171,"significance":"If the claims hold, the pipeline and dataset are a modest but useful step toward automated BPMN generation, addressing a real gap: AND gateways are severely underrepresented in the PET dataset. The paper reports exact F1 values and explicitly acknowledges limitations such as the CatBoost flow-relation F1 of 62% and difficulties with implicit gateway closures, which is commendable. The manuscript also states that code and datasets will be made available through GitHub, which would support reproducibility if the link is provided. However, the significance is currently tempered by an evaluation design that does not yet establish the central parallelism claim: the main comparison is uncontrolled, the gold standard is author-created with no inter-annotator agreement, and the pipeline test documents are not described in terms of provenance or holding-out.","major_comments":[{"comment":"The RQ2 claim that adding LESCHNEIDER 'significantly improved' AND-gateway detection rests on an uncontrolled comparison. Baseline (a) is a 5-fold cross-validation on PET alone, while Model 3 (c) is a 5-fold cross-validation on the combined PET+LESCHNEIDER corpus; these columns are evaluated on different test folds, so the 0.00-to-0.23 B-AND F1 improvement conflates added data volume with added parallel-structure content. The symmetric ablation needed to answer RQ2 is missing: hold out the same LESCHNEIDER folds and compare a CRF trained on PET only with one trained on PET+LESCHNEIDER. The one available cross-corpus condition, CRF Model 2 trained on all PET and tested on all LESCHNEIDER, yields 0.00 B-AND F1, indicating no transfer of AND-gateway knowledge from PET to LESCHNEIDER. Without per-fold scores or intervals, the reported improvement cannot be distinguished from split noise, so the central parallelism claim is not yet established.","section":"§5.2, Table 3"},{"comment":"The LESCHNEIDER dataset is both the training enrichment and the evaluation gold standard for RQ2, and it was authored and annotated by the same team with no reported inter-annotator agreement. Because the texts were written to include parallel structures via explicit markers such as 'and simultaneously' (§3.2), a model trained on part of LESCHNEIDER can match the annotation style of the remaining part without learning a generalizable notion of concurrency. The paper should report annotation guidelines, inter-annotator agreement, and ideally evaluate on an externally produced test set; at minimum it should justify that the manually drafted markers reflect naturally occurring process language. This is load-bearing because the dataset itself is the paper's main contribution.","section":"§3.2"},{"comment":"The pipeline evaluation does not state the provenance or holding-out of the six test documents. If these documents are LESCHNEIDER texts that were used to train the NER model, the reported element F1 of 89.2% and relation F1 of 73.7% could be in-sample estimates. Please specify whether the six documents come from PET, LESCHNEIDER, or a separate set, and confirm that they were excluded from all training and hyperparameter selection. Without this information the headline pipeline numbers cannot be interpreted.","section":"§5.4, Table 4"},{"comment":"All F1 values are point estimates from five-fold cross-validation with no per-fold breakdown, confidence intervals, or significance tests. Given the tiny positive class for AND gateways (8 in PET, 40 in LESCHNEIDER), a B-AND F1 of 0.23 could easily arise from sampling noise. Reporting per-fold scores and an interval or significance test would materially strengthen the comparison and is necessary to support the RQ2 conclusion.","section":"§5.2 and §5.4"}],"minor_comments":[{"comment":"The paper reports '32 new AND Gateways' while Table 2 lists 40 AND Gateway mentions; please clarify whether these are gateway elements or mentions and reconcile the counts.","section":"Abstract and §3.2"},{"comment":"The text 'For the CRP' appears to be a typo for 'For the CRF'.","section":"§4.2"},{"comment":"The metric formulas are formatted as 'PrecisionPrecision=...' and would be clearer with standard equation formatting.","section":"§5.1"},{"comment":"The title contains an extra space in 'T ext'; please proofread the final text.","section":"Title"},{"comment":"Table 1 notes that the PET dataset was updated to version 1.1 by the authors, but the paper does not specify where this updated version is archived; please provide version information and a repository link to support reproducibility.","section":"Table 1 and §4.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable fit for a specialized BPM/NLP venue, but the central RQ2 claim currently rests on an uncontrolled comparison over an author-created gold standard. I would ask the authors for the controlled ablation, dataset documentation (including inter-annotator agreement or external validation), and clarification of the pipeline test-set provenance before considering acceptance. The novelty is incremental but acceptable if the evaluation is tightened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"If you work on process extraction from text, this is worth a skim. The new LESCHNEIDER dataset - 15 documents with 40 AND-gateway mentions and deliberate noise - is a real contribution, and the full pipeline (NER with BERT/RoBERTa/CRF, RE with CatBoost, coreference resolution, diagram generation) is a reasonable engineering effort. The authors are also honest about limitations: they report the CatBoost flow-relation F1 at 62% and note problems with implicit gateway closures. The BERT-base-cased result over larger models is plausible given the tiny dataset. The soft spot is the central RQ2 claim. Table 3 shows B-AND Gateway F1 going from 0.00 (column a) to 0.23 (column c), but those are different test sets. Column (a) is PET-only cross-validated on PET; column (c) is combined-data cross-validated on the combined corpus. A model trained partly on LESCHNEIDER is being evaluated on LESCHNEIDER folds, so it can match the authors' own writing style without learning a generalizable notion of parallelism. The only PET-only-on-LESCHNEIDER condition, column (b), scores 0.00 on B-AND. To support the claim, they need PET-only and PET+LESCHNEIDER models evaluated on identical held-out LESCHNEIDER folds. They don't report that. The positive class is also tiny (8 vs 40 mentions), and there are no confidence intervals or significance tests, so 23% could easily be noise. The pipeline evaluation in Table 4 is on six test documents with no stated provenance and no error bars. And the paper says the code and datasets are in a GitHub repository but gives no link, which makes the work effectively non-reproducible right now. Taken as a proposal, the paper is fine: new annotated data, a standard but sensible pipeline, and an honest discussion. But the headline empirical claim needs a proper ablation, external validation on text not written by the authors, and availability of the data. I'd send it to peer review, but the reviewers should push for a redesign of the RQ2 experiments and a dataset release.","headline":"A useful small dataset and pipeline for BPMN extraction, but the headline AND-gateway improvement is not supported by a controlled comparison.","tokens_in":654,"tokens_out":1284,"would_cite":false,"duration_ms":39453,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 15-document extension to the PET corpus raises AND-gateway F1 from 0% to 23% and gives 89.2% element F1 in the full pipeline.","keywords":["BPMN generation","process extraction","parallelism detection","AND gateway","PET dataset","LESCHNEIDER dataset","named entity recognition","relation extraction"],"falsifier":"Take a held-out set of process descriptions from unrelated domains that express parallelism with connectors other than 'and simultaneously' (for example, 'while', 'in parallel', or 'at the same time') and compare B-AND Gateway recall for the combined-corpus CRF. If recall drops to near zero, the measured improvement is an artifact of the dataset's phrasing rather than a general parallel-structure capability.","tokens_in":10046,"feed_emoji":"🔀","tokens_out":8560,"duration_ms":84964,"temperature":0.7,"pith_summary":"Automatically converting process descriptions into BPMN workflow diagrams is hard because existing training data contains few parallel (AND) gateways. This paper claims that adding a 15-document, 32-gateway dataset called LESCHNEIDER to the PET corpus teaches models to detect parallel structures: the F1 score for B-AND Gateway rises from 0% to 23%, with precision at 50%. On six held-out documents, the full pipeline reaches aggregate F1 scores of 89.2% for elements and 73.7% for relations. If this holds, small, parallelism-focused dataset augmentation is a practical lever for a task that mostly stalls on writing-style variation.","feed_headline":"15 new documents lift parallel-step F1 from 0% to 23%","feed_subtitle":"A small new corpus teaches workflow models to detect parallel steps, pushing element F1 to 89.2 percent.","key_machinery":"The central object is LESCHNEIDER, a manually authored extension of the PET dataset: 15 process descriptions, 91 sentences, and 32 AND-gateway annotations built by drafting BPMN diagrams from real-world sources and then writing text that follows the Seven Process Modeling Guidelines. It carries the argument by rebalancing the AND-gateway label distribution and by providing textual markers such as 'and simultaneously' that the models can learn for parallel structure detection. Around it, the pipeline combines token-level NER (CRF, BERT, or RoBERTa), a CatBoost relation classifier over mention-pair features, coreference-based entity resolution, and a directed-graph renderer that outputs Business Process Model and Notation (BPMN) diagrams.","core_discovery":"The paper's central claim is that the LESCHNEIDER dataset, 15 manually drafted and annotated process descriptions containing 32 AND gateways, fixes a critical data imbalance in the PET dataset (8 AND vs 117 XOR gateways) and thereby enables BPMN extraction models to recognize parallel splits in text. In the authors' experiments, a CRF trained on the combined corpus improves B-AND Gateway F1 from 0% to 23% while most other labels keep their performance, and BERT-base-cased is the best NER model because larger models overfit the small corpus. The assembled pipeline, using CatBoost with random over-sampling for relation extraction and coreference resolution for entity merging, produces BPMN diagrams with aggregate element F1 of 89.2% and relation F1 of 73.7% on six test documents. The authors also report that the flow relation (F1 62%) and implicit gateway closures remain the main bottlenecks.","pith_inferences":["Because the 15 LESCHNEIDER documents were written by the authors, the reported gain may be tied to the specific phrase 'and simultaneously'; parallel connectors like 'while', 'in tandem', or 'at the same time' should be tested to see whether the improvement generalizes.","The paper reports no inter-annotator agreement for the new annotations, so an independent re-annotation of the same documents would clarify how much of the F1 gain is signal rather than a single annotator's style.","A natural next experiment is to keep the pipeline fixed and vary only the training corpus, replacing LESCHNEIDER with an equal-size corpus of parallel-heavy documents from a different domain; the B-AND Gateway F1 delta would measure dataset value versus corpus size."],"forward_implications":["Models trained on the combined PET plus LESCHNEIDER corpus should detect AND gateways in unseen texts, where PET-only models currently score 0% F1.","The LESCHNEIDER additions make parallelism a measurable, trainable target instead of a rule-based afterthought.","BERT-base-cased is the recommended NER backbone for this data size; larger BERT and RoBERTa variants overfit and should be avoided unless more data is added.","The full pipeline can serve as a reproducible baseline for end-to-end BPMN extraction, with element F1 around 89% and relation F1 around 74%.","Relation extraction, especially the flow relation at 62% F1, is the current bottleneck for accurate diagram generation."],"supporting_citations":[{"why":"Supplies the original PET dataset with the 8-vs-117 AND/XOR imbalance that LESCHNEIDER is designed to fix.","marker":"[12]"},{"why":"Defines the BPMN tagset, the CatBoost relation-extraction setup, and the PET imbalance figures that motivate the new dataset.","marker":"[3]"},{"why":"The Seven Process Modeling Guidelines followed when drafting the LESCHNEIDER diagrams before writing their textual descriptions.","marker":"[21]"},{"why":"Rule-based approach whose 'and simultaneously'-type markers the authors mimic to create AND-gateway training examples.","marker":"[4]"},{"why":"Second source of conditional markers for detecting AND gateways, justifying the dataset's parallel phrases.","marker":"[22]"},{"why":"Provides one of the pre-trained NER encoders compared in the study; BERT-base-cased is selected as best.","marker":"[7]"},{"why":"Provides the larger pre-trained NER encoders whose overfitting on the small corpus is used to justify the BERT-base-cased choice.","marker":"[8]"}],"fun_headline_variants":["15 docs with 32 gateways lift parallel F1 from 0 to 23","New dataset boosts BPMN parallel detection from zero to 23% F1","Adding 15 annotated texts makes BPMN models spot parallel steps","Parallel gateway F1 jumps from 0% to 23% with small corpus fix","How 15 documents taught BPMN extraction to recognize parallel splits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 15 hand-written LESCHNEIDER documents, annotated by the authors with no reported check that other annotators would label them the same way, are a valid sample of how real process texts express parallelism.","fun_headline_variants_meta":{"raw":{"variants":["15 docs with 32 gateways lift parallel F1 from 0 to 23","New dataset boosts BPMN parallel detection from zero to 23% F1","Adding 15 annotated texts makes BPMN models spot parallel steps","Parallel gateway F1 jumps from 0% to 23% with small corpus fix","How 15 documents taught BPMN extraction to recognize parallel splits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000233,"raw_usage":{"total_tokens":1476,"prompt_tokens":913,"completion_tokens":563,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":529,"completion_tokens_details":{"reasoning_tokens":461}},"tokens_in":529,"tokens_out":563,"duration_ms":6591,"temperature":1.0,"reasoning_tokens":461,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:20:16.774890+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out set of process descriptions from unrelated domains that express parallelism with connectors other than 'and simultaneously' (for example, 'while', 'in parallel', or 'at the same time') and compare B-AND Gateway recall for the combined-corpus CRF. If recall drops to near zero, the measured improvement is an artifact of the dataset's phrasing rather than a general parallel-structure capability.","supporting_citations":[{"cited_title":"PET: an annotated dataset for process extraction from natural language text tasks,","cited_arxiv_id":null,"evidence_quote":"Supplies the original PET dataset with the 8-vs-117 AND/XOR imbalance that LESCHNEIDER is designed to fix."},{"cited_title":"Beyond rule-based named entity recognition and relation extraction for process model generation from natural language text,","cited_arxiv_id":null,"evidence_quote":"Defines the BPMN tagset, the CatBoost relation-extraction setup, and the PET imbalance figures that motivate the new dataset."},{"cited_title":"Seven process modeling guidelines (7pmg),","cited_arxiv_id":null,"evidence_quote":"The Seven Process Modeling Guidelines followed when drafting the LESCHNEIDER diagrams before writing their textual descriptions."},{"cited_title":"Process model generation from natural language text,","cited_arxiv_id":null,"evidence_quote":"Rule-based approach whose 'and simultaneously'-type markers the authors mimic to create AND-gateway training examples."},{"cited_title":"Aconceptforgeneratingbusinessprocessmodels from natural language description,","cited_arxiv_id":null,"evidence_quote":"Second source of conditional markers for detecting AND gateways, justifying the dataset's parallel phrases."}],"review_version":1}